I find the awkwardness of saying no comes from what is unspoken.
A - would you like to come to this thing with me could mean:
1 I would like to go to this thing and I’m looking for someone to go with as I don’t want to go on my own.
2 I would like to go to this thing, I think you would too, I’m doing a nice thing for you
3 I’d like to spend time with you/get to know you better, here’s a suggestion.
If it’s 1 then you can just say it’s not your thing knowing that they’ll ask someone else.
If it’s 2 they can be a bit surprised or offended that you don’t want to do it. Either they don’t know you as well as they thought (you don’t want to do the thing) or worse, you just don’t want to do it with them.
If it’s 3 then they’re not that wedded to the thing itself but would like to do something with you.
Safest answer for all scenarios is. I’m sorry, that’s not really my kind of thing but I’d love to get together with you. Do you fancy meeting for [alternative thing]?
Unless you don’t particularly want to see them or they get stuck on scenario 2 and persist in being offended. I’ve had real problems with both of these