I have to disagree with the person who said nervous is dangerous - not always the case. Confident and aware is much better than nervous, but over-confident and aggressive is far more dangerous than nervous - most drivers like that live round me.
OP, I often hate driving as it can make me nervous. The more I drive, at different times of day, the better my nerves are. I would really recommend this. Also, talking to myself when I drive if I'm having a nervous day helps me. I have a CD in my car with all my favourite songs on which relaxes me a bit - I think I associate the music with being more relaxed, so it initiates some sort of Pavlovian response and I do relax - every little helps.
I think the endless pressure to become A Super Awesome Confident Driver and believe Isn't Driving Super Fun etc is just stupid and weird, yes to a degree when I've been driving a lot over a couple of weeks my nerves fade and a bit of enjoyment creeps in, but that's normally if I have to do a lot of pre-7am driving - it's never been the driving that's the problem, it's other drivers. When they're all safely in bed not driving into other people, it's fun to an extent but when I really want to treat myself to a nice day out, I get the bloody bus.
Don't put yourself under pressure. Try to do a bit more, and try to cover an unfamiliar road or roundabout or junction once a week/fortnight, perhaps with supportive company in the car. You don't have to love driving, it's not possible for everyone to love driving, that's just a load of crap. But you don't have to feel like you're being hunted down by other drivers, or something's going to go wrong, or you're doing it wrong, or what have you. I would recommend you work at it until you are sure you are a safe driver, you trust your instincts.