I did kickboxing and mixed martial arts as well as jujitsu & brazillian jujitsu recreationally on a 6-day a week program for 4 years.
During training I sustained 2 concussions, about a year apart. I have given up completely (even recreationally) despite wanting to do it as my main career choice and loving it, because of this exact reason. I never lost conciousness either - HOWEVER, I do honestly believe at this point in time you are alright.
While I think you may have just had your bell rung, and no permenant damage is done yet (everyone gets knocked around in fights, and if you had no ongoing symptoms) your LONG TERM health could suffer if you continue to partake in the fights. Getting hit in the head definitely isn't good. While I do support contact sports completely; I think the athletes should definitely know when it's time to hang up the gloves.
From your post, it really doesn't seem as though this is a sport, which does increase your chances for getting hurt. However, listen to your body. If you feel like it's time to ask this question - maybe the question should be if you should continue.
As it stands, no, I don't think you would see long term negative effects (though I'm unsure of how many 'quite a few' is)... but be careful. It does sneak up on you and you don't want it to do that. CTE and dimensia and other neurological problems are very real if resulted from multiple head injuries. Boxers call that pain and blurry vision "punch drunk"... that's not a good feeling, and pain is a sign something is wrong. I get it - it's just pain, toughen up, take it and dish it out. You become best friends with it when you fight. But you have to listen to your body when that fight is over, or even during the fight if you have the ability to walk away (though I realize in real life scenarios that may not always be a safe option either).
Long story short - if you play it careful, and be totally honest with how you feel, you will be okay. As it stands now, I'd be confident in saying that you're okay, so don't worry. If you don't feel like it's a good idea to continue, there's probably a reason for that however; and your body/mind could be trying to protect itself before something bad actually happens.
But if you were to continue on a regular basis over a long period of time, in the future you may begin to see the result of problems due to multiple head inuries, even if it is just getting your bell rung with no ongoing symptoms.
This is definitely only based on experience and what trainers/doctors have told me over the years; and is not a diagnosis. If you're worried, you can have an MRI done to check for any physical damage already done; but you shouldn't wait until the damage is already done to stop.
I think you're okay, but definitely be smart & honest with yourself. Good luck. :)