I would stand back from this and assess. He sounds a shit. First cheating, and then compounding the fact by showing you once and for all that he's not just sexually untrustworthy, but a snake in other aspects of life?
You could confront - and probably sort this quite easily:
'No, I don't want a break. I'll come along. You always say I should have come along afterwards so why don't we just cut out the middle bit, forget the 'just for the boys' thing and all go as a family, eh?' Then in private: 'You need to stop doing this - the cutting me out. It's not helping, DS has noticed, and it's embarrassing - it's so obvious that you're trying to buy affection - what you can't see is that doing it by trying to cut me out has the opposite effect. DS has seen you disrespect me, and therefore our family, now he's just seeing you do it in a different way. It's the same result. Either you're for our family, or you aren't. And if you want to stay married and in the family at all, leaving me to pick up the shit doesn't help either, and also, that's not exactly invisible to DS too.'
- but do you want to? You could stamp out this behaviour, and it's unlikely that your DS is going to be able to be bought by someone who clearly doesn't have the real loyalty to the family to provide the security that you do - but at the end, you'll be left with this as the person you grow old with when your DS flies the nest. Are you sure about that?
Use this behaviour - it is telling you a huge amount. If this is his MO, you're simply going to have this issue time and time again with other things - money, friends, every time a change in your lives comes along and you have to field the challenges brought by the fact that the person who should be standing with you, undermines you to put himself in pole position. Needless to say, it shows more than anything that his cheating wasn't a one off 'mistake', but a real reflection of a selfish, throw-others-to-the-wolves mentality.
I would really assess whether I really wanted to be with him, and I'd make plans and enquiries as to how life might look without him.