It's very complicated. Like nigels, I do think that contact in some form should be preserved until they are of an age where they can make their own decision.
Also, surely if he is having a breakdown or some other form of MH issue, then his current reaction to them is unlikely permanent. In that case, I personally think that at some point in the future he would really regret losing contact with them.
As DD1 finds the visits very stressful, it would be worth exploring other options for having contact with him. If the situation becomes solely about preserving some kind of relationship with/bond to him rather than him assisting with childcare to give you a break, then it seems possible he could do that in a way that DD1 finds more positive.
I would ignore the hospital letter distraction and go back to the dialogue about the girls, especially as you raised it. Can you not say to him - look, what I am trying to establish is whether you genuinely want to spend the with them or whether you are doing it solely out of duty.
DD1 is struggling to cope with the change and I am trying to figure out what would be in her best interests...? Or something along those lines.
If the girls have a positive relationship with other members of his family, it would be a real shame to lose that. I say that as someone who has very little family. I literally only have relationships with my parents and more distantly with sister (she lives far away).
Out if interest, what do your mum and dad think about your idea of cutting contact?