Why do I think he might be mid-affair? Because he's treating you and the children really badly, provoking arguments and becoming disproportionately angry and stressed. This is a more proactive stance than pre-affair, when the behaviour then is more reactive and defensive i.e. you complain about something and he reacts defensively instead of apologising, perhaps counter-attacking that you are nagging. Behaviour when it is a full-on affair would be more akin to this morning's episode, creating a horrible argument out of nothing at all. This is guilt-ridden behaviour, whereas pre-affair it was "find me a justification" behaviour - and no real guilt.
Really have a think about what you're saying - that you want it to run its course. If he really hasn't done anything yet - and your instincts on this are better than anyone's - stopping this now does not mean it will drag on for years, or that he will be regarding it as an itch not scratched. Far from it, he will look back and feel eternally relieved that his wife had the good sense to stop him from doing something disastrous. If you can get him to admit that he nearly crossed the line - and you're able to prove that's all he did - it doesn't stop there you know.
He's done this before - he is someone who, rather than resolve relationship difficulties in an adult way, is unfaithful. If he never dealt with that character flaw after his first marriage ended, it has remained firmly implanted.
He might have told you that he was unfaithful because his relationship (or his wife) was crap - and like a lot of people, both you and he believed that, thinking that it would be different this time.
But unless an infidel digs deep and examines why, regardless of their relationship difficulties, or the awfulness of their partner, they chose deceit over walking away or better still, dealing with their unhappiness, the problem remains - and it is no surprise at all when they enact the same behaviour in subsequent relationships.
So this time - he needs to deal with it. If he does, this won't be haunting you for years at all. He might just emerge at the ripe old age of 50 something, much wiser, more self-aware and a committed monogamist.
I meant to ask earlier, when did he last mention your weekend away in March?