What he's been through is terrible, without a doubt. But just because he had a fucked up childhood doesn't mean he gets to do what the hell he wants at the expense of everybody else's feelings. Because he's suffered doesn't mean he gets to make you suffer.
My father had a very fucked up childhood, very much in the general ballpark of your DHs. Because I was young and hadn't had therapy yet, my first love and LTR was a man who'd had a similar upbringing involving maternal neglect and time in care. We'll call him G.
G abused me because, at his heart, he had a strong dislike and distrust of women; he was angry with his mother but couldn't take it out on her so he took it out on the women who loved him. My father was similar until I was in my 20s and he had a complete nervous breakdown and ended up in psychotherapy, but much of the damage had been done to his wife and children (all of whom, unfortunately and ironically, were women).
Me and my Dad have a good relationship now because he did the work and I had therapy. But G and I didn't last because he didn't do the work, wouldn't have therapy and was just furious and lashing out. He was also utterly faithless; he punished women by cheating on them, it had been a pattern from his first relationship at 16. We split after eight years together when he was 36, almost 20yrs ago. I would be willing to bet he's still repeating the same patterns and will do until he dies or reaches breaking point like my Dad did and literally has no choice.
I repeat how I started this @rosa1983; because he's suffered doesn't mean he gets to make you suffer. You love him and if you want to you can support him, but only if he puts in the work and only if it's what you want to do. You have every right to walk away, whatever he's been through. He's had a terrible time but that doesn't give him cart Blanche to do whatever he likes and have you there waiting for him at the end of it. He may be someone you love, he may be the father of your kids and his mother may have really hurt and abused him, but what he's doing to you now is not okay. You owe it to yourself and your kids to give yourself some distance and take stock of what you want to do next.
Your DH needs help. Deep down he almost certainly knows that. But you need help too and I'd suggest you get some therapy of your own to help you (and your kids) deal with all of this.
Good luck with everything my lovely, wishing you so much strength.