I also ask if they want to do stuff when I have my son at the weekend and it's normally we are tired, your dad's hungover
Do you only have your ds on the weekend OP? It sounds like your DP already have regular plans for weekends.
Do they help out during the week?
I have my GS 2 or 3 times a week for my DD to work. I take him to messy play, the park, woodland walks, soft play, toddler groups etc We also spend time indoors playing. My DD asked me if I'd go to a Pumpkin Patch with them last weekend. I didn't fancy it tbh so I said No. I've also refused to go to a Halloween Party with them this week. I love my GS dearly and love spending time with him. I will, readily, help out when needed. However, my life no longer revolves around my DC or their DC.
When you reach a certain age, and have made sacrifices to raise your own family, parents need time for themselves, even if it is just time to relax in front of the TV, spend quality time with their DP or socialise with friends.
I know in advance when I'm having my GS and push everything else aside to concentrate solely on him. I don't see the need for me to tag along to toddler parties. His mother is perfectly capable of taking him herself. She usually finds other parents with similar age children to talk to and new friendships are made.
It's not usual for GP's to accompany GC and their parents everywhere.
When my GS starts school I will be happy to watch him in his school concerts and pick him up from school for tea before his mum comes home from work. I don't want to be with him 24/7.... That's his mother's role (When she's not working).