Adult survivor of alcoholic family here. Lots of experience from childhood with drunk people, not just the addicts but also the other friends/fam who drank socially. From nearly 40 years dealing with drunk people, this is what I can tell you - drunk people never lie. They can be shitty, shameless, destructive, cringey, horribly insulting, terrifying, even sweet and funny but... they cannot lie.
Whatever drunk people articulate when they are drunk is the truth they'd normally never say - or wish they had the guts to say - when they are sober.
Sorry OP. Even if you "forgive" the current man in your life for making a "drunk mistake", you will never be able to keep him for more than a few weeks. It's just a matter of time. Once his ex calls him back or if he even meets someone like his ex, he will ditch you AND your kids, and disappear.
I know it is tough to give up on a potential partner, but the pants don't fit, you know that yet you are thinking, "no no if I keep ignoring it or believe in the best, it will eventually fit." It won't. Sooner or later, your association with this man will implode. Think carefully if you really want to risk it and put yourself - AND YOUR 4 KIDS - through this shetfest, for what, two days a month?
Taking tough but right decisions - That's what being an adult, and being a parent, is about. Don't walk into quicksand - AND TAKE YOUR KIDS WITH YOU - even after knowing it was there all along.