If you read through some of my past posts, you'll see that has been my primary symptom for 3+ years.
Doesn't matter if I feel anxious or not. Sometimes I catch myself taking quick, shallow breaths; other times everything seems normal before I get hit with it.
For me, it is Vocal Chord Dysfunction caused by anxiety. I have had my heart, lungs, nerves, etc. all thoroughly checked and its always clear.
Definitely possible to feel short of breath all day and not actually 'feel' anxious (outside of the whole not being able to breath think
Xanax as needed, and talking to a psychologist have helped me a ton. Best of luck, and God Bless!