Whether or not you should go is up to you really. You know yourself better than us, so in that regard i'd say "If you think you'd regret not going, you probably should." But I'd like to offer a bit of advice that helped me.
If you're afraid you'll have anxiety issues, just tell yourself you will definitely have anxiety problems. That there's no question you'll feel bad. Do that and a lot of the fear of not knowing should go away; worst case scenario is still slightly better than not acknowledging it, since you will be at least a bit more ready for it if it comes. best case scenario is you proved yourself wrong.