Yes, YABU. Try a little compassion. Not every woman, or person, who is afraid of driving, is batting their eyelashes and waiting for a man to save them. Driving is dangerous by nature - why do it if you don't feel competent?
I am 52 and only recently got over my paralysing fear of driving. I was in a very bad accident at age 16, but then managed to drive all over the US, including on motorways, in my 20s. It was the small roads I was afraid of as I had been in head-on collision.
Then I came to live in Europe, didn't drive for several years, and had a terrible time starting again (in a big city, capital of a country whose drivers are known for being aggressive and unpredictable). It made life very difficult for me when my child was small. Even with good public transport, shopping and getting anywhere in the suburbs can be a nightmare.
I decided a few years ago that if I didn't get over my fear soon, I never would. I had CBT, which didn't help, and finally just forced myself onto the motorway. It was terrifying at first but I finally realized it was just fine and now would be OK driving anywhere. I do have a bad sense of direction but try to plan trips carefully in advance and have a printed map in addition to the GPS.
And I can think of at least four couples my age where the man does not drive. Ever. either never learned or doesn't want to. What do you care what other people do anyway?