^Don't is more assertive when dealing with judgemental, nosy feckers. Basically.^
This.
Usually it only comes up in conversation if it's for a specific reason:
"Let's all have a trip to insert-out-of-reach-by-public-transport-attraction with the kids, who's up for it?"
"I'd love to but I don't drive".
"Ooh, we could have a trip to IKEA/out of town superstore and pick up XYZ"
"Good idea but I don't drive"
"I'm getting a taxi to the restaurant/pub/cinema if anyone wants to join."
"Not driving then?"
"I don't drive."
Most people won't ever have noticed you don't drive until you have to point it out to them. And then they're incredulous and sort of patronising and act like you're not a fully functioning adult and how on EARTH have you managed and well... it gets old. So you say the thing that is guaranteed to shut the conversation down as quickly as possible.
In my experience, non-drivers hate asking for lifts. We may accept them if offered and truly convenient, but sometimes it really is more convenient to just go ahead with your well-practised plans for getting there under your own steam.
I can never understand why other people get so bothered by whether or not someone drives a car. It's not inconveniencing them 99% of the time.