I hate this too but disagree with those who say you are stuck. You can say no to things!
It depends what it is. If it's a friend who asks if you're free and then suggests paragliding or something and you don't want to do it, you can then say it's not your thing.
If it's someone who needs you to do them a favour such as take them to Ikea to pick up furniture or to catsit or look after their child or something, it's then really awkward to wriggle out of it. What are you then supposed to say? You're forced into either going along with it or causing a massive drama by saying "No, I don't want to do that", or "No, I'm not up for that, sorry".
It's rude and manipulative of the person asking. The only people I know who do this are asking for something not that exciting or for some kind of favour. It's deliberate - so that you can't use the "sorry, I'm busy" excuse later. They are then extremely offended when you say you don't want to/can't do it. And can use the "but you're free/not doing anything else" statement to push you into it.
I'm a musician and I get people ringing up asking if I'm free on such and such a date with no explanation. I might or might not want to do the gig for various reasons. I might be technically "free" on that date but need to time to rest or to practise for another gig. When I was starting out I just used to say "yes, I'm free" and they'd then say "oh, great, blahblah" and it would often turn out to be something that I really didn't want to do. Difficult then to back out.
However, I learnt pretty darn quick.
"Are you free on such and such a date?"
"I'll have to check my diary. What's it for?"
"Blah blah"
"Ok, I'll get back to you on that by the end of the day"
Go away, "check diary" (which is all in my head anyway), decide what to do and then phone them back.
Also gives me a way to say no if I decide that they aren't willing to pay enough for the time and expense it will cost me.