This is a common symptom of both anxiety and depression. Think you will find that nearly all sufferers do the exact same thing. You are far from alone on this one. I always say to just try and enjoy their company today. In ten or twenty years time you can look back and see all the time you wasted worrying over them, that you missed out on spending good time with them. Time without worry.