I struggled to learn to drive. It took me two years of lessons and I failed my test 3 times. Someone I know told me that I obviously wasn't up to it and to give up. Eventually I passed on the 4th go but the examiner told me that it was very close and I'd scraped the pass. After I passed, I still kind of felt like I didn't know what I was doing and carried an internal belief that actually, I'm an inherently bad driver and someone will probably get hurt if I'm on the road.
Two months later, I moved abroad to a city with excellent public transport so I didn't need to drive. The years went past and with each year of not driving, my already low driving confidence got lower and lower. Then I was a passenger in a car when we crashed, and will never forget the sudden, unexpected impact. Confidence down again.
I started having to drive again 10 years after i passed my test and would be in tears as I drove, my biceps would ache from gripping the wheel and I couldn't even sit properly on the seat as my always poised to break.
I've been doing the same short trips for 5 years (work, the grocery shop and train station) and while im not on the verge of having a panic attack each time, i still get spooked very easily. At one point, the thought of being back behind the wheel was terrifying, so I feel like I've made some progress.
If I see a diversion sign I completely panic. If I have to edge into a line of traffic or change lanes, I feel like I have almost zero understanding of space and have a very limited understand of time/distance. I find it incredibly difficult to know when to go at a roundabout and if someone keeps at me for anything, I feel absolutely devastated and it's like confirmation that I shouldn't be driving. The level of panic I feel when I drive outside of my safe routes is not only terrifying but also absolutely humiliating. I've had strangers park my car because something happens and I panic so much I literally can't think.
I always worry that I will kill someone and not only will it be my fault, but the added shame of having done so while knowing it was 'likely', would be insufferable to endure.
I've tried CBT. I don't live somewhere where there is Passplus as I'm abroad. It's a terrifying phobia and I do my absolute best to push myself and overcome things without ending up a crying, shaking mess.
My husband does most of the driving. It makes me so sad.
I hope this perspective is insightful for you OP so you can have an idea of why someone who has a car won't drive. Terror.