Not suprised he thinks looking four children alone is a piece of piss, it sounds like he's only does it occasionly and for short times, and because it's rare he does it, during those times he gets to take shortcuts like takeaways and lazy days in front of tv, I can imagine that would be easy, but if he's not doing the majority of stuff alone day to day, he won't get how stressful it can be.
The occasional few hours he has them he won't be experiencing planning, shopping and cooking meals for everyday, he won't be making sure they have clean clothes, he won't be the one booking appointments and planning those around school and commitments, he won't be having his head filled all day with planning, sorting, organising, arranging, doing, cleaning everything for everyone in the household, so of course he thinks it easy.
He either thinks you experience day day things the same as he does on the few occasions he's been alone and fairies do all the other stuff. In which case he needs a sharp shock.
Or, as I suspect, he knows very well how hard it is which is why he's going when they are meant to with him, and knows you'll be there to do the parenting he's opting out of for the weekend.
I know you said you don't want to not see the kids for a whole weekend, but honestly, I'd be taking equal time out, I'd be planning it for when all four children are there, and wouldn't be doing things like making sure he doesn't have to cook, or wash clothes or plan stuff for kids while he's looking after them either. You don't have to go away on the piss, just a few nights in a hotel alone. If he genuinely thinks it's easy he won't mind or complain.
It's not him going out/away with mates that's the problem, it's his attitude that has to be when and where that's convenient for him and you should find it easy and stfu because he's the important one. He's not willing to compromise, like ask mates to go away when he doesn't have his two children. He won't change if you keep doing everything for him though.