OP,
Don't be forced into anything dramatic.
Simply make a list of replys that work for you.
That doesn't suit.
We have plans.
That doesn't work for me, etc.
If pushed, just say something like "I have enough going on with my own children, I really haven't time to be minding other".
If they try and argue the point, don't reply further.
You don't have to discuss further.
If your parents try and say anything, tell them to mind the children.
If he's such a bully, it is likely the children will pick this up.
I wouldn't want them around my children.
Spend time with other children.
Fade them out.
You don't have to let on you're upset or annoyed.
You don't owe them an explanation.
You are just choosing to no longer oblige them.
End of.