Idk now. Domestic abuse from women is relatively rare, wrong, but rare. How long ago was this?
He was hardly just out of a marriage, as he didn't really have a marriage in reality, did he, as he was unfaithful just a few months before taking vows and then it was over six weeks after. I also wouldnt have taken the view that he did 'right by her' by marrying someone having a child he fathered. Very poor decisions indeed, and his fault. I am not sure why this woman has had such a detailed part of your current relationship, when it was so brief.
Whats happening with the baby now, presumably a young adult now?
He hasn’t responded very well from me being hurt my all of this, in fact he has punished himself for it.
I don't like the sound of this at all.
Your reaction, wether that be hurt, or anger, or distancing, whatever you do is your reaction, and he cannot control that, but, he is affecting your reaction by his, which is very worrying.
Your relationship is very unhealthy. You haven't known him long.
I don't think you know him very well.
Whether he seems kind and gentle and all those things, doesn't obliterate some of his behaviours, including marrying so quick and blaming the woman for it all.
You are too close to see the wood for the trees, this is a common hallmark of earpy on in abusive relationships. I'm not saying it is abusive, but its not healthy for either of you from what you say its quite toxic. Toxic relationships are full of this type of hurt and pain, and 'reactions' to your valid response to his disclosure.
You cannoy heal him and he cannot heal you. Neither of you sound ready, but it definitely doesn't sound good for you OP, and relationships are supposed to be, and freeing, not so intense and constricting with fears for his behaviours.