So he cheated on her twice that you know of.
He left her for you and then a year later texted to see if she would take him back.
No wonder the Do right by me and my son reply. That makes sense now.
She lets him stay, and he holds up the phone to her when you argue.
I tell you why she lets him stay, because she knows it'll get at you, revenge possibly.
He clearly has self esteem issues, to cheat and to be so dismissive of those that care for you only stem from not caring enough about yourself to prize those that do.
He doesn't deserve you. He knows it too. he is a low down cheat.
when you have an affair with someone elses H, there is always a risk that they will betray you too. This is kind of what he's doing now.
I hate OW, but I think given your past relationships, you were vulnerable and not really ready for a full on, full time relationship, so an emotionally and physically unavailable man followed on from your ExH, and you didn't have the strength to hold out for better.
End this relationship, get some counselling to deal with the terrible blows to the esteem and confidence you have taken and use the next year to rebuild yourself and heal the hurt you have suffered.