OP I just wanted to share my own experience because this thread inspired me to really do something about my driving. I would love to be able to use it as a bit of a support thread!
I could have written so many of the other posts. I learned to drive in my late twenties and my driving instructor always said the only thing holding me back was my confidence. He didn't put me in for my test until he was pretty sure I would pass first time, and I did, although I did spend a fairly long time getting there. I knew I could so easily lose that confidence again and told myself that I would have to get straight out there and drive after passing. However, my circumstances then changed in several big ways in a short space of time, including moving to a totally new area. I put off driving, and I felt with all the stuff going on in my life and the new unfamiliar place I could justify putting it on the back burner. But then it just dragged on and I knew deep down that I was only making it harder but I was frightened so I avoided it and then the stuff I'd learned in lessons just felt further and further away, and I then felt I was more of a liability. I eventually did start driving a bit but it was only ever in fits and starts and I was never a "proper" driver. At my best, I could go a few short, known routes. And I felt so proud of myself! At my worst I would go months barely going anywhere and would have to really talk myself into the tiniest drives, even to places I had previously driven without any worries.
I don't think I'm a bad driver as such, and I don't feel massively anxious when I'm actually in the car. But, for example, where I live everyone has unofficial parking spaces. However, if I go out and someone has a visitor, they will park in my spot. So if I'm out somewhere, it's always on my mind that I might get back and find I can't park, and I don't have the confidence to just go and park on the road in case I don't do it properly or can't get into a space! I haven't parallel parked since I passed my test! This is the kind of "unknown" that makes me avoid driving.
Anyway, when the OP started this thread I read all the replies and thought "I've got to do this now." I haven't done masses, but I've been out several times with DH and a few times alone with DCs and it's an improvement on not driving at all! What I have found best so far is going alone, early, so I can get familiar with driving without the pressure of loads of cars on the road. Then going at busier times with DH so I just have that reassurance if I'm a bit unsure on something. It's usually knowing when to overtake a bus or a cyclist or something and I haven't ever been wrong but just value having him there to confirm that my own judgment is fine.
The thing that gave me a bit of a boost was to ignore where I thought I "should" be and be realistic about where I actually am. Yes, I've had a driving licence for a fairly long time now. And I definitely have built up some valuable experience in that time, which helps me be a better driver. But I've also lost a lot of knowledge because there are several things I used to do in my lessons that I've just never put into practise since, driving short distances in my local area. Instead of dwelling on this I've asked DH to basically give me refresher lessons, without any preconceptions other than I clearly once was good enough to pass a test, but clearly 7 years of not practising certain things will leave me a bit rusty.
Really hoping one day I will come on this kind of thread and say "This used to be me but I got over it and now I drive all the time!" Best of luck to all those in similar positions!