I used to do think that way. When I was eating a sandwich I'd eat around where my fingers were touching and I'd leave those little pieces on the plate. I did this until my entire family noticed and started bugging me about it. I guess embarrassment fixed my situation, but germaphobia is really not an embarrassing problem - a lot of people have it. Even google celebrities with the fear of germs; it will surprise you.
All I can really say is that I work at a hardware store - Home Depot. I get my hands dirty all the time, handle lumber, bags of cement, and numerous types of garden-related tools. I also eat with my hands all the time, often forgetting or being too lazy to wash them. While I do think this can increase your risk of catching a flu or cold, it is not harmful unless of course you're handling potassium cyanide or arsenic beforehand (I kid). Clearly any sort of poison or faecal matter should be washed off, but simple dirt is relatively harmless from my experience. Obviously use precaution - a good hand washing regimen is a solid step towards being healthy, but germs can attach to your hand through an almost infinite means, so aside from washing hands when you eat, your risk of catching some heinous or deadly virus really is slim.
Some of my clients come in with concrete-caked hands, or dirt-crusted fingers, and I see them picking their teeth, their nose, and eating things with their hands all the time. They're fine, so are you. :)