There was a huge thread on this a few weeks ago, with some very odd generalisations and assumptions (I'd say borderline accusations) about both drivers and non-drivers, so apparently it's a polarising topic!
My H can't drive (medical reasons, but it's an "invisible disability") and it never occurred to me to mind. (And thank goodness it didn't!) But he has received some very strange/rude comments, mostly when we lived in the USA. We don't have or need a car now, and I prefer not to have one. We do rent one occasionally on holiday (his parents live on Skye and it's a bear at night or on holidays if you have to take the bus or bike or taxi there, although he does it when he goes alone) and of course I drive in that case which is fine - there are other things he's better at.
This comment is downright odd to me: I think it shows a lack of independence, organisation and responsibility. I've found it takes a lot more of these qualities to get around without a car on a habitual basis in a lot of places I've lived or visited. Plus you miss out on so much life cruising around in your own private little cannister! Someone else could just as easily say that driving/owning a car when you don't need to is lazy, selfish, and myopic.
All of that said, it's your subjective decision. If you live or plan to live in a rural place with a bunch of children, you might absolutely rely on a car and it could be annoying if you're the only one in the household who can operate it, especially if you don't really enjoy driving. So, YANBU if you feel it's a dealbreaker for you, but YABU to the idea it's some universal or even usual dealbreaker.