If you are a family that has a routine (which I glean from "we do my stuff in the day" remark), and he knows that, and there is a plan for his mum that evening, and the fact that he asked you if it was OK, all points to it NOT being OK.
If you usually just have a bunch of daffs and some burnt toast, and then get on with your day as usual, then it would been fine. But you obviously have a tradition of sorts, and therefore YANBU.
Is it possible that they haven't realised the impact, in which case he could easily have said "we're busy, can I come over Saturday", but if they are fully aware of which day it is (and also have a tradition of fuss about mums), then it sounds as if it is form on their part too.
Agree with pp who says you've been put in an impossible position. I sit in that position quite often. Seemingly lovely requests from DH "is it OK if I help out my sister/BIL's cat" which can only be answered with "Of course my love, I wonder and marvel at your sense of decency and capacity to help all around" when the end of the sentence should be "and yet you can barely remember to put the bins out in our own house, and our cat would starve before you noticed to feed it".
I have vested interest in the matter, perhaps.
Ideas:
1 - Ask if he can help them on Saturday
2 - Ask him if he can have a nice pub lunch with them in the middle of putting up summer house/trampoline/retaining wall fun, make the booking in the evening your evening out with your kids and him.
3 - Ask him to consider Saturday your Mother's Day, as you clearly won't get any particular joy/space/sleep on the Sunday as you had been expecting.
4 - Tell him you'd be delighted to see him spend all day with his parents, and in a few weeks time (when he thinks it's all gone fine) have a light but clear conversation about future expectations (But please do try not to sound like a mumzilla/screaming banshee because it is so unbecoming when we have the expectation of others considering us and our feelings).
Good luck!