I understand that some people can't drive because of health reasons, but apart from that, I can't fathom why people wouldn't want to learn. Most people are anxious and nervous about it, so that doesn't seem like a valid reason to not learn IMO.
And as for 'I can't afford it!' Excuse me, but most people can't afford it! Do these 'I can't afford it' types think everyone had free bloody lessons, or got taught by their dad, took one test and passed, and had a car given to them for their 18th birthday FFS?! MOST people can't afford to pay for lessons and tests, but we still do it! 
And I also wonder how some women (who depend on their DH to drive,) would cope if he dies/leaves. Driving is such a great life skill to have, like swimming, and even riding a bike. All non-essential, but very handy.
If you don't want to learn, then don't, as long as you don't expect lifts, or for people to go out of their way to pick you up/drop you off. And I'm sorry, but many non-drivers do this, even though many deny it.
@53rdWay said she is always being offered lifts and people insist on giving her lifts, but no-one I know - including me, would ever insist someone has a lift. In fact, I rarely offer one, unless I am going the same way. I won't go out of my way for anyone.
I paid a lot to learn to drive, and I pay a lot to run and maintain my car, so I don't see why others should benefit from it because they can't be bothered to drive. Funny how some people say 'I care about the environment, I prefer to walk, keep fit, get the bus and so on,' yet they will quite happily take lifts, and even expect you to go out of your way for them.