Thing is AleC, sneaking around having an illicit affair is one thing, quite exciting and romantic I imagine. But now it's no longer that, it's a day to day relationship with all the usual boring stuff to deal with, finances, kids, practicalities...so his previous escape from reality has been replaced with a new reality. On top of knowing how he's treated you and his kids. Add to that the isolation from his friends, your family, perhaps the disapproval of some of his family, and it's obvious why he's a bit snappy.
Also your marriage grew, built on a solid foundation, you shared the mundane stuff from the start. These two muppets have only had the fun up until now, there isn't the background, the foundation to build on, so of course the cracks will show sooner rather than later.
They will probably put on fake smiles and pretend it's all great, as my ex and his OW did. But people who know them will see through it, and even more maddening for them, they will know people can see straight through it.
Also remember that he justified leaving by telling himself that he deserved happiness and it couldn't be with you. You, quietly coping, building a happy, secure home for your kids has blown his myth right out of the water. Friends and family are full of admiration for the strong, loving woman and mum you are. They are looking at him, shaking their heads and thinking, 'silly, silly man.' By seeing you coping, he is facing exactly how much he thrown away. My ex was bitterly angry with me for being ok, for coping. But it's what we do, we have no choice.
He will, in the next few months wake up and realise what he's done. And he will probably, like my ex, become even angrier with you. Why? Because it reflects his own stupidity and selfishness.
You'll be fine, your kids will be fine. There is no doubt about that.