"My rule of thumb is if you're asking if you need to go, then you don't. You'll know if you have to go."
That could be for a majority of the time, even if you are clinically diagnosed for anxiety but not always the case. I've had several instances where something came up and I just kept asking my self and others do you think this is anxiety (was burning throat and thought it was acid reflux from anxiety) Turned out to be strep.
For the OP, if you arent allergic to any headache meds you can try those to see if it subsides. If not then it doesnt always hurt to try urgent care.
My doc always said if you are unsure if something is real or just anxiety, you do a process of elimination of what you know, known OTC treatments and have had tested or been told.