Has your DH ever been affectionate?
I mean before the affair?
I think that maybe it was a dealbreaker for him, but he won't let you go for some reason.
Even though you cheated, you are free to end the marriage if it's not working for you.
If your DH is saying that's just not him, then you won't get your emotional needs met by him and if it WS me, I'd seperate and divorce.
I won't be held to ransom by threats of suicide.
You need to decide what is right for you. You don't need his permission to leave. There's no point in hoping for something that won't happen.
Maybe when he realise you aren't going to stay in the marriage as it is, he'll try and improve. If he does fine .... if not then you move forward.
Some people just can't get passed infidelity. That's fine ... but there is no point in being miserable. I find men especially, just staying married, but full of bitterness and resentment.
BTW - you can do long term fostering which is very rewarding and it pays. Even if you leave the marriage, you can be a single parent foster carer.
I presume you considered adoption or decided it wasn't for you?
At the end of the day, you only get one life. Don't be miserable.