oh yes, i get this all the time. it's especially nervewracking when i'm in public or when i'm driving. it doesn't make me nervous at home because i can just lie down in bed and wait it out, but if i'm out somewhere then i get awful thoughts of passing out or blacking out or a multitude of other awful things (none of which ever happen).
i've found that one way to combat this fear of feeling groggy is to just ask myself, "does anyone without anxiety get nervous at feeling groggy?" the answer is always no, and that sort of pulls me out of my anxiety and makes me see it from a different perspective; knowing that feeling tired/groggy/out of it is a normal sensation that everyone goes through, but us folks with anxiety blow it up into something much bigger.