"It's not a huge inconvenience...."
But it IS an inconvenience. I too got into something like this and I ended up with a bagful of pre-planned excuses which I pulled out in a random order so I could break the expectation. The person in question rarely expects lifts anymore. They still do ask (well, ask would imply a 'please' but they stopped years ago) but it's a 90% fail rate for them.
Five extra minutes in the opposite direction to home every single fucking time is annoying. It costs you. It is extra time, extra fuel, extra wear and tear to the car and extra effort. And it costs the non driver nothing.
It was actually my DH who put his foot down and told me to come up with excuses to put our pisstaker off when he realised that we already had and were still spending a large amount of our hard earned money to a) learn to drive, b) pay for the tests, c) buy the car itself and d) pay the tax, fuel, insurance, MOT and upkeep of the car whilst the non driver could spend their cash on nights out, tattoos, smoking and basically blowing cash they could use to become drivers themselves.
OP: to avoid unpleasantness, make up excuses and break this habit. A single excuse would be handier. Driving to an ailing elderly parent straight after every session or something like that would work.