It sounds like things are very fractious between you both.
I don't think it would be fair to put the children in a position to have to 'choose' but equally grandparent relationships are also very important to kids so I don't see why you'd need to change the arrangement as it stands provided they're happy with going to their granny's.
When you say they're happy with it do they actually look forward to going or do you mean they just aren't miserable with it?
Do they look for more time with/ express missing their dad? Do they look forward to going to stay with him? Does he care for them well and has there been a history of abuse either towards you or them? Have things been amicable between you before now?
I think op obviously navigating seperation and Co parenting is extremely difficult but I think you need to be really careful that you're centering your kids in it all. They should be in the place where they're happiest, well looked after and with a stable consistent care giver. If that's their dad in comparison to your mother then it's something you need to give thought to from a neutral place putting your own feelings aside.
However, if the kids are genuinely happy with the current arrangement then no, I don't think he gets to try and dictate where they spend time when it's on 'your' time. The caveat to this op, is that you then don't get an equal say. So if he gets a new girlfriend who you don't like and he leaves the kids with her you've given up your right to argue that outside of abuse.
Is there no scope to work together to agree that any time either of you need to use childcare that you'll ask the other first and use grandparents as the backup to that? Or do you think his intentions are malicious?
How much time are the kids needing to stay with your mum and how regularly is it happening?