I sympathise with you, Fairbags! I don't drive, spent a lot of money on lessons and failed several tests some years ago now. I admit that I've never found anything so difficult as driving, just didn't come naturally to me so I've sort of drawn a line underneath it and don't drive.
As a family we get lots of comments because my dh doesn't drive either, (passed his test years ago but just didn't enjoy it), and we have two daughters. However, we both live within 15 mins. walk away from our work, (the children's school is even closer)! Also due to the nature of my dh's job, he's entitled to a company car which as we obv don't use, we get a yearly allowance in lieu, which we use for transport. People look horrified if I say we're taking a taxi and start insisting we have a lift with them, but we just say we're not spending out on a car, so we expect to pay for our own transport. We'd hate to rely on people and never have!
As a family we're all very comfortable with buses, taxis and trains, have positioned ourselves in a central area, and even when we go on holiday we'll pay out for a return taxi to the airport, have done this for years!
We have had comments from people and also they can get a bit offended if they offer lifts and we decline. If we've pre-booked a taxi or there's a very convenient bus home from somewhere which stops in our road, (as most do around here!), why put somebody else out?
People have hinted that our dd's must miss out on out-of-school activities, but they don't. Public transport is so good around here, plus anything local they're very keen to scooter or cycle to! We don't feel our lives are restricted at all, but I suppose it's a case of what you're used to. People don't comment so much now, maybe they're more used to us too!