I think if you've never driven, you can develop a sort of blindness as to the extent you're relying on others, especially if you don't take lifts on a regular basis -- just occasionally, as an exception. For the first decade we were together, DH couldn't drive, and we were too broke to own a car. We both used public transport or walked, and he'd swear blind we hardly accepted any lifts. We're just not car people! You can manage perfectly well without! [pats self on back]
But we relied on lifts every time we visited family in Devon. We rented a car (I drove) on holidays in Cornwall. His parents always insisted on picking us up from the station in their city, because we'd come so far already and they didn't want us to spend another hour on the bus. People gave us lifts between rural churches and wedding reception venues; to funerals; from the station to the holiday cottage we'd rented with a group of friends in Yorkshire; to pick up Freecycle furniture; to catch ludicrously early budget flights. People gave us lifts when I was heavily pregnant and had SPD.
It wasn't one person doing all of this driving, it didn't happen every month, and people were always kind and insistent in offering to help -- but, over a decade, we accepted so many favours without being able to reciprocate. It embarrassed me. Non-driving DH was oblivious. Obviously, he'd say thanks, but it never bothered him, especially as most of the lifts were relatively short. It's not a big deal, is it? And people always offer! Surely they don't mind.
Then we did get a car, and DH dragged his feet about getting a licence. As far as he was concerned, he commuted by bus to work, we walked to the supermarket, we didn't need to drive. Apart from all the hundred-and-one exceptions when we did, in which case I was on hand.
If lots of people in your life are actively encouraging you to learn, even offering to pay, I'd take the hint. A phrase like '...if OH is at work, I use public transport' suggests that your partner probably does ferry you (and 4 kids) around a fair bit, whenever you need to go somewhere beyong cycling distance. Being the only driving partner becomes very frustrating, and it's not just the driving; it's being solely responsible for MOTs, insurance, tax, AA, repairs, oil changes, tyre pressure and the other tedious details.
Obviously, it's completely different if a person is medically unable to drive, or neither of you can afford to run a car. But not having 'any interest' in driving is pretty entitled, if you do have access to a vehicle.