I don't like to drive too much as my judgement is poor when sleep-deprived, and I have spent most of the last 4 years sleep-deprived. I live in London and only drive if I've had a reasonable night's sleep. I am not very confident on motorways, but I have friends outside London who do lots of driving but refuse to come into London, so it is partly a question of what you're used to.
Also I passed my test then for various reasons (moved away from the car I was learning on, no money etc) didn't drive for about 4 years, so was an out-of-practice new driver when I started, this too has had an impact on how I see myself as a driver.
If I had to drive for my job, as DH used to do, I'm sure I'd be more confident, but the basic fact is that if I get behind the wheel tired I am unsafe (in my opinion). Not driving much has effected my overall confidence and DH now does the driving on long trips - something we used to share. He loves driving, whereas it has always made me tense.
If I needed to drive every day (maybe if I didn't live in a big city with most things in walking distance or with great public transport) I like to think that my attitude to driving would change quite quickly. But for the moment I'm happier walking and getting the bus, using the car occasionally and having DH drive us for long drives.
I wonder how many confident drivers have something else they feel nervous about. Baking? Swimming? Speaking a foreign language? We all have something.