You were not responsible for protecting his fidelity.
He is full of BS. If he had issues in your relationship, he had several ethical options to use to deal with them, including communicating with you, couples counseling, or ending the marriage. He chose infidelity. He wanted to pursue OW, so he did.
While you were at home caring for your vulnerable DD and the other children, he was out living a wild party lifestyle like a single man. Instead of being your stay and support, he let all of you down massively.
He is not truly remorseful. Yes, he was full of self-serving drama initially, but his anger, blame-shifting, lack of empathy and proaction, and rejection of your recovery requirement show that he is not committed to helping you heal and restoring your trust.
There are more secrets. For one thing, he is minimizing his relationship with OW. She expected him to leave you for her. This suggests that they were conducting an EA prior to having sex, with most of their interactions occurring during the work day and nights out (since you say they had limited phone contact). He had mentionitis and commented how funny she was and how well they got on. He was flattered. They likely bonded emotionally and he confided in her. He would have told her things about you, your marriage, and the children. I don’t believe for a minute that the purpose of his several calls to her after their assignation was to express guilt. This liar is lying.
Something triggered his sudden confession, and it wasn’t remorse. Perhaps OW or someone else in the know has been in touch and he panicked. Or perhaps this is an exit strategy.
Whatever his agenda is, he needs a sharp consequence. You don’t have to tolerate his toxic presence or his drinking.