I sometimes see it as a sort of socially acceptable 'learned helplessness'.
Men have 'can't operate the washing machine' and women have 'can't drive at night'.
Or a sort of attractive vulnerability thing? Like wearing high heels?
SIL didn't learn until she was in her 30s and BIL has to drive everywhere, she tries to pretend she walks places in the rain to save the planet but I think mostly it's because she doesn't want to drive unless she absolutely has to.
My mum stopped driving at about 50, because she kept bumping the car (minor scraping the curb, opening the door into a post stuff). Possibly the 'higher earning man' dynamic of the past, but she saw it as having bumped 'his' car. Whereas if my dad bumped the car he'd just get it fixed (or not) and not even think.
Again, BIL and SIL share a car and he is the high earner, so...
On the flip side though, my brother can't drive, and I don't think he ever will, he's 35. I think with him it's a fear of failure. My ex also couldn't drive, just never learned (lacking ambition, that one). I don't think it's as socially acceptable for men though, so more of them push through the dislike.
I have astigmatism and though annoying, I don't think it makes driving at night dangerous, I've always had it so I'm used to it. That said, I do massively prefer unlit roads at night.
I would also never share a car with DH, I might borrow his (and drive it verrrry carefully, he does like his things to stay shiny), but I prefer to have my own which can get dirty and muddy and a tiny bit battered (scratched wheels from bumping up a curb to let an ambulance by, etc) without having to be accountable to anyone else for the state of it.