If you want your husband back, you have to go counter intuitive.
At the moment hes come running back to you and told you everything because its the least worst option.
What seems to be happening now is you are siding with him against her.
Although this is natural and understandable, dont lose sight of the fact that he did all this.
He absolutely is capable of doing it again, especially if there is no substantial consequences for his actions.
He hasnt lost anything at the moment, he is probably shitting himself thinking 'phew, she hasnt kicked me out, i may be ok here'
OW has every right to tell who she wants, this is her life, she probably feels used and betrayed, even though she may be unstable and dramatic, your husband still decided he wanted to fuck her.
Like others, i think it has been going on longer. At least emotionally.
Your husband is doing everything he is doing to save his own skin, and you are assisting him in that.
Clearly she is not a safe bet for a relationship, if she was, you may not have been told and he could well be making plans to leave.
He is not on your side, or 'in' the marriage in the way you need and want. Start seeing who he really IS not who you want him to be.
Other more experienced posters have suggested you kick him out, this is to make him realise the fullness of the consequences of his actions as well as give yourself some space away from his sorry (he got caught) ass and think about what you want.
This is yours and your childrens life now, thanks to him.