I can relate as well. I don't like driving, to be honest, and wish I didn't need to. Usually I'll do what you do: worry about it beforehand. But, when I get behind the wheel, I focus more on the actual driving, so even though I'm still anxious and my heart's pounding, I don't pay much attention to it at all.
The best way to deal with it is to simply drive; drive short distances to start with, and work your way outward when you are comfortable. It doesn't have to be much.