It's rubbish that this necessarily means he's having an affair. (Of course, it's a possibility, but it is by no means the only or even the most likely explanation).
OP, I know (vaguely) a woman who is in your shoes, though her children are a bit older. Her husband works in a high-powered, high-stress job. They have two young children. He quite often stays out until very late, drinking with colleagues - I would guess at least once a week, I know it is often more. I know this because I am an ex-colleague, and my DH and I have been there with him on the odd occasion. (We normally go home a bit earlier than the 4am that he can drink til though). His drinking buddies are often other men with children who have a pass for a night out. Only they do this occasionally, and he does it every week.
The problem is not another woman - the problem is that he is basically quickly becoming a functional alcoholic. He didn't want children, fell in love with fatherhood when they arrived, and then took on more responsibility and more stress to pay for the family to have nice things. He is a lovely bloke and very good at his job, but it does involve networking after work, so you can see how the slide into this pattern started. 
I truly believe he still loves his wife and kids deeply, just not perhaps the domestic work. Which is crap of him, because he also made the choice to have children. I think in his head, the fact that she is a full-time mum and he is earning to pay for everything possibly legitimates his absence. I am not sure it really does compensate, though.
I don't know if any of this rings a bell with you, but maybe a conversation about work/life/financial priorities is in order. 