In your post, you indicated that your physician put you on the antianxiety agent, Buspirone, but there was no indication about any form of therapy that you have engaged in. I just started cognitive behavioral therapy (CBT) a few weeks ago. CBT teaches you to face your fears. It is considered the therapy treatment of choice for anxiety. I suggest you look into that.
Your problem reminds me of the Serenity Prayer: "God, grant me the serenity to accept the things I can't change, courage to change the things I can, and the wisdom to know the difference." There is nothing that you can do to change your past, but there is a lot that you can do to change your present situation, and that will ensure you a brighter future.
The very best to you, Am!... Chuck