I hear you, OP. I have been the same since the menopause. Even talking about driving makes ne cry. The last time I drove anywhere to meet friends I had to turn round & drive home again after half an hour because I was so stressed about the thought of driving home later that I couldn't enjoy the meal.
I used to have to drive a child to a club once a week- it was only a 2 mile journey but I would be thinking about it & in tears from the drive hone one week to doing it the next.
I can walk to work/everywhere& live near good transport links. If I can't get somewhere,on my own then I don't go.
If I could only ever drive at 4am on a June morning I'd do it but I can no longer cope with the unpredictability. Just because I drove to Aldi ok yesterday, what might be on the road today? My processing speed is slow & I am terrified of causing or being involved in an accident. Consequently I sm hyper vigilant about every little thing which makes it worse.
I've had refresher lessons- ok with sn instructor but useless on my own as I don't trust myself. Can't go eith DH as we'd be divorced before the doors were shut!
I think I certainly have ASD traits. I feel completely ashamed& embarassed thst I have been unable to overcome this anxiety & am a complete failure as a person - it's tantamount to saying you can't read or ride a bike or swim.
But OP you know yourself best & if you are happier not driving & it doesn't impact anyone else......then ignore the other people.