He blocked this woman the night I saw the messages of his own accord
I'm so sorry OP - but no he didn't.
He blocked them because you had found him out.
That's not of his own accord - it was a kneejerk response to being caught.
and he did it infront of me.
Performative nonsense, as you have now established that he unblocked her behind you back.
He's a selfish arse, & it's a brutal way to behave ... so ... you need to get your own brutal head on.
From now on, make every decision only from the perspective of what YOU want & what is best (or least worst) for YOU. do not attend to any declarations, promises, regrets, apologies, or justifications from him.
I felt like I had to give him another chance, especially with kids involved which is why it's blown my mind that the contact is still there!
You don't have to, & neither are your children a marital sticking plaster.
He can continue being their dad, no matter what you decide, but you do NOT have to continue being his wife.
That decision is SOLELY down to you. Please focus on that, because you need that autonomy & personal choice right now. You do not owe your H another chance unless you genuinely want to offer one.
Have a browse round ChumpLady's site.
She is an astonishing source of support, wisdom, & unfailing snark.
www.chumplady.com/about-chump-lady/
You don't need to make any immediate decisions, & you don't need to listen to a word your H says right now. If you can take a few days away with family or friends, so much the better. I am sorry he's been so cruel & selfish. Take your time, & do what's right for YOU. If that means doing nothing for a while, that's ok too. You have a lot to process, & don't need to jump any guns to please any of us posting here, ok? This is YOUR life, & you get to choose whether your H is qualified to be a part of it any longer. That is up to nobody but you. 