I have the same fear as you. About two months ago I thought I was exposed after being inside a storm drain tunnel and seeing a bat fly around. Got the vaccine, was over my fears for about a month, then I got scared the vaccine gave me rabies. Got over that one because if the symptoms I thought I was feeling were actually from rabies I would be dead now, and they went away anyway. Not I have two new sources of rabies fear. About three weeks ago I got a very small splinter in my finger, and that same finger still hurts. Even after seeing the splinter I still think it was an animal bite that I did not detect (sooo stupid). Another one I have now is when my aunt's dog licked my foot, and I just happened to have a small cut there from a few days before.
The reason I posted my experiences in your thread is to show that you aren't the only one with these craxy irrational fears. All three of my experiences are just so stupid when I think of them harder. For example, the first one: I never felt anything touch me yet I was still afraid that I was somehow exposed. The second one: I think I would remember an animal biting my finger, especially if it was something like a squirrel, rat, or rabbit. And I even saw the very small splinter in my finger that was causing the pain, and still is today, even three weeks later. The third one: all dogs need to have their vaccines, and the rabies vaccine is one of them. I can also find out if the dog had rabies, because if it did, it would have symptoms within ten days. (I'll be putting this one to rest soon, but I'm sure another equally ridiculous fear will come up between now and then.)
Now I'll go through the same process with your fear. First you need to know that the only way to get infected with rabies is to get the saliva of an animal that has rabies in your blood/mucous membrane. You never felt the squirrel touch your hand/fingers, and I doubt you would have touched the same place the squirrel was eating from. If you were picking up a sandwich that somebody had taken a bite out of, would you pick it up by the side where the bite was, or on the side of the sandwich that did not have the bite? I think the side that did not have the bite. Now apply that to the banana. Would you touch the part of the banana that the squirrel had been eating? I don't think so. That's the only place that the saliva of the squirrel would be, if there was any saliva from the squirrel on the banana at all. Think of the last time that you took a bite out of a banana, or any other food, how much saliva was left at the place that you took the bite from? Not much. Now think of how small a squirrel is. They have much less saliva then humans do, and we don't leave much saliva on our food after taking a bite. Next, the rabies virus is very fragile. It can not live outside of the host for very long at all. That's why almost all transmissions to humans are from bites. Because in a bite, the teeth of the animal are making direct contact into your flesh. The virus was never really out of a host. You were not bitten. Next, squirrels and other small rodents are the least likely animals to carry rabies. Because of their size, it doesn't take long for the rabies virus to kill them. And during the shot period that they have the virus and are alive, they are very susceptible to being eaten by other animals because they are in a weak, diseased state. Even if the squirrel that you were feeding had rabies, which is very unlikely, the way that you think you were exposed is not a way that you could catch the virus.