A quick favor is rarely ever quick. You say it that way because it minimizes the interruption it has on your day.
If you literally live next door, and you literally have nothing to do and you're sitting around twiddling your thumbs and your friend calls you up and asks you to help with a 20 minute job, you'd be happy for something to do.
It sounds like you have a much further commute to get to your friend, and the job is never quick and easy it always drags out as such things tend to do.
I've gone over to a friends for a quick favor before, a job that literally could only take 20 minutes but they can't do on their own, but I don't drive home after that. It's 50 km to get to their house, it's 50 km to get back, once I'm there, I'm going to have a cup of tea or we're going to go out for lunch or we're going to go out and do something together or we're going to have a visit, yes it's true that they may have said they needed me for a quick 20 minute thing but it does end up eating up my entire afternoon and I don't love that.
Once in a while, for something that's really important that they can't wait to get resolved and don't have anyone else that can help them for, well, I'm happy to help, but you got to keep in mind that I only have so much free time and you're taking it away from me. That's how I feel