@wannabemum38 of course you're going to feel that way. I feel that way and haven't gone through the heartache and losses you both have. The health of your baby and ultimately yourself will naturally be your priority and when baby is with you they will always come first.
I can also imagine that could be hard to hear for your partner though, esp when you've gone through so much to get to this point - although he should also understand why. His reactions just now - could they be him protecting himself? He could be terrified of this pregnancy not going to term because it's something neither of you have control over but he can't feel what we feel, Which in turn, is not allowing him to enjoy your stages pregnancy together, it's harder for the partner to feel as connected as we do I guess, because it's our bodies that change and they just feel the same, so almost feels unreal to them in a way.
Have you guys had any counseling individually or together around the pregnancy and what you guys have been through? (You don't have to answer that) I've found it odd, in todays day & age especially, how they still only ask us how we are feeling and coping psychologically but they never ask the partner. if you've not done counselling together around everything maybe it could be an idea.
The reason I'm thinking this is that From what you've said, this could be a protective mechanism he's built dating from his childhood when he doesn't know how to handle his feelings and as a result he could be disassociating and in return shutting down and becomes emotionally unavailable to you too. It doesn't mean he doesn't want the baby, he might not know himself how he's feeling at the moment so doesn't know how to articulate that.
If he won't talk to a doctor/professional, maybe see if he'll try journaling, where he can mind dump his thoughts and feelings each day then close the book. You can't read it though, but usually it's the starting block and maybe he'll start talking things through with you more, these things take time though and also take time to even try as it seems very woowoo esp for men, but it might help him realise his feelings as a beginning step.
I do hope it's more he needs additional support himself over the idea that he's being an arse.