Yes it sounds like anxiety to me. My Mom had COPD and she only got out of breath when she got up and moved around, but she was fine just talking.
Do you find that your breathing is shallow from anxious tension? Perhaps you aren't breathing correctly. I'm no expert but maybe doing breathing exercises would help you breathe more deeply so you aren't out of breath.
Take care :)