I had more or less the same thoughts of people judging me based on my appearance. As a result, I would become quite anxious. So anxious that, just like you, I would begin to avoid social situations altogether. Big mistake. I felt even worse in my seclusion, wallowing in self-pity. And, of course, I just wanted to go out, meet and talk to people.
The solution? I don't know. We may have similar obstacles in life, but we still live in different circumstances. Personally, I just learned to shrug it off. People will be people. Everyone makes judgments of others to an extent. We're trying to identify others. It's what we do. Now if someone doesn't take kindly to me, then screw em. They can keep staring. I'm talking to someone else.