I'm sorry your doctor wasn't as helpful as you would have liked. I've also had that talk with my doctor recently, and it seems that just on principle, she never wants to be overly comforting. Doctors generally don't want to tell anyone that they 100% don't have a disease. The best they can do is tell you what your doctor did -- that you are not showing signs of anything dangerous. Try to take that as a good sign.
I have had MS fears myself, even recently (numb hands and feet). For some reason, I have been able to overcome that fear because I know so many people living with MS. My husband's 80-year-old mother, who only started to need a walker a few years ago. A friend's husband, who has some vision trouble in one eye but otherwise functions perfectly fine (his vision loss was instantaneous, not gradual as you are describing, although I have no idea what the normal presentation is). A former co-worker, who runs marathons and has kids and lives a full life. For me, I realized that MS is not necessarily something to be terrified about, even if you do have it. Of course, nobody wants to have it, and I understand the fear.
If you feel like you need additional peace of mind, there's nothing wrong with going to a neurologist. Don't feel guilty about going back to your doctor and asking for the referral that they offered. You also might find some comfort in seeing a therapist. I've been going to one for over a year now, and although I'm still having anxiety, it's nice to have someone to talk to who doesn't judge you. Get a referral for that too if you can. Medical doctors are not always the best at saying the right things to people with anxiety disorders. Psychologists / counselors / therapists are much better at that. But you should take comfort in that it sounds like your doctor really doesn't think that you have MS.