yes, it's annoying and common. All of a sudden you realize that you've not been having anxiety issues. This is the worst possible thing you could do, because then you start thinking about it. And because its been relatively easy, and because you rapidly start thinking too much about it, it tends to cause rather explosive panic attacks.
It happens much less over time though, and is shorter and less intense when it does come. Eventually you get "used" to it and it doesnt affect you as much. Usually. One of the reasons I come here is so I never let it get too distant from my mind.