Have to say I agree with NN here LL. It's easy for us to say when we're not going through it, but how you respond to your DH's contact with his children will determine long term how happy you are with it and therefore also, how happy they are.
His smirking is a pita. But he smirks because he knows he's wound you up. Don't let him know that, and then he'll have nothing to smirk about.
If the contact time is 4 hours, arrange to do something in those 4 hours slightly more interesting than housework - something that you can't do while the children are there. That may be getting your hair done, nails done, swimming, running, even DIY that's too dangerous to do (tools) while the kids are around. That way, you might be miserable while they're away, but at least when they get back, you'll have something to show for the time they were away.
Try to see him taking the children off for 4 hours as a slot of time for you. If it's longer than that - say a day, 8 hours or so, arrange to do something more adventurous - go and see one of your friends who live 2 hours away and tell your ex you're doing this and will be back at x o'clock.
At this stage, don't start getting angry with him about bringing the children back late. But you could ask him to talk about the practicalities of feeding them etc. That makes it a reality for him that taking them off is not just him giving them a treat, he's actually got to look after them properly as well. And it also shows him that you are genuinely accepting his right to take them off with him, like it or not. And if he knows you accept that, he's much more likely to be co-operative with you and not mess you about, if he's a half-way decent person.
BTW, I don't think there's anything wrong with him spending time with them and his family together. Now if he'd dumped them on his mother while he went out with the lads, then I'd think that would be off - but he didn't, he stayed with them in a family setting. Nothing wrong with that.