Yes it is a mess indeed.
Let's separate things out a bit.
- You are ok for your children to have a relationship with dad
- You're not concerned for their safety
- You would benefit from time to your self
- You are ok with them having a relationship with your family...?
- You're not concerned about their safety with your family
Are you financially independent since the split? And it sounds like the children are financially set for the future - is all that secure?
Curious about the financial set up between GP and ex but sounds complicated. Is there a link between them providing access to the children and their financial position with him? Sounds like it?
If all the above is right, then your problem is that the kids don't want to go and it's distressing.
In which case as previous OP, stop making the teenager to go, and let the others. Explain when they are older they have a choice too but for now they don't.
The reality is that it sounds like the children aren't at risk. They might not like it but then loads of kids don't, and at least they are seeing family members at the same time who they do want to see (right?). So I think could be a lot worse.
Could the Grandparents to school pick Ionian Friday if they don't want him to? If they're being made for this, make them work. They sound narcisstic and manipulative.
In fact through all of this your problem is mostly then. You have no grounds to remove them though, restraining order?