I know only too well this feeling, and your are not alone. I have suffered Anxiety attacks, and agoraphobia on and off for 20 years or more. I had a job until 3 years ago when I was made redundant after 10 years in the same job, the fact that I lived where I worked helped a great deal, it was like my home, my safety net. After this period I had an interview for a job, and before, during and after the interview I had panic attacks, it was awful. I got the job, it wasn't far from where I lived about 5 minutes walk or so, but every day I was there I had panic attacks, I worked in a kitchen as a chef, so felt when I was there I couldn't escape, because sometimes it was just me working - it truly was awful, it made me very depressed and my anxiety worse I just didn't realize how difficult it was to work again. I had to leave the job, I felt like the anxiety had really beaten me, it was a very sad time, I remember going to church one day in tears, and just asking God, if he was there, and why is this happening to me, if life is going to be like this, then I didn't want it. I was arguing with my partner, as she just didn't understand what I was going through, and thought I was just being silly, that also makes it so much worse when people are so unsympathetic toward you, I was literally at the end. I split with my partner, and moved a few hours away back home, it was hard but I got over it. I got another job, but this time I took it slow, and got a menial job, something that had no real responsibility, and part time, to ease myself back. When suffering from this abhorrent, and destructive disease, you have to literally rehabilitate yourself from scratch, and fight it, I never fought it before, I allowed it to consume me, but now I am more clever, Yes I still suffer, but inch by inch I am regaining ground. Take your time, do something that is not stressful, or with to much responsibility, something you can manage, start of part time, Something close to home if you can, you still will feel anxious, I did, it didn't just go away, but it wasn't nearly as bad as before.
The best of luck, and believe me, there are many of us like you, we are not weirdo's or odd bods, we are just suffering from an illness, that is no mean feat to overcome.