This is not the place to ask this as so many people here have been in the receiving end of great heartache. But I will try.
Life is not always straightforward and people behave badly, have affairs when they should have had the courage to end it first.
Men are usually the worst for this behaviour not solely because they are trying to stay married and shag on the side but SOMETIMES (not always) because leaving an unhappy marriage also means leaving children they genuinely love.
It's not the right way to behave at all.
However it happens and trite comments about 'leaving a vacancy when marrying your mistress' are neither helpful nor necessarily true. It depends entirely upon the person and the nature of the previous relationship.
It is salient to remember that second marriages have much greater staying power than first (45% for first and 34% for second end in divorce) so the evidence doesn't really bare that saying out but naturally makes an injured party feel better.
The focus has to be the children. The most important thing you can do at this stage is to get some humility and not thrust your new found happiness upon anyone but each other in private. Let the devastated children (and the abandoned mother and father of you have left a partner) come to terms with their new lives by making sure you both do your utmost to make the practicalities of their lives as easy as possible in the circumstances.
Don't be dicks about contact.
Don't be dicks about maintenance.
Don't introduce any children to anyone until you are sure it's going to last. At LEAST six months.
If you have children DO NOT MOVE IN WITH HIM WITH THEM.
His children have to adjust to him not being with mum and at home first. He needs to get his own place and establish a routine of having them there on a regular basis. Without you anywhere in sight.
They don't need to have Dad leave AND dad have a girlfriend in their lives all at once.
See him when your kids are with their dad.
You started at a moral low point. You have a chance to try and do things better. If he really is 'the one' he will want to do this bit a lot better. Slowly and quietly is the only way.
He needs to stop being a wanker and tell his wife. This bit is despicable.