"I am worried I am going to loose my partner if I do not drop my defence but I am really worried that he is not behaving responsibly and I don’t want to be at such high risk? Am I being to careful or am I doing the right thing?"
No, he's not behaving responsibly, not at all. He's behaving as if there is nothing going on in the wider world and is choosing not to protect himself (fine, his choice) and by extension choosing not to protect those who he comes into contact with (not fine, not fine at all).
Your situation is that you are the primary carer for your daughter, and it would be very disruptive and downright distressing for her were you to get ill. He is not the primary carer of anyone - that's a big difference. You also are considering the welfare of of your 95 year old grandparents and wish to stay in contact with them. He isn't considering the welfare of any of his children or his parents.
"Any thoughts on what’s best to do?"
Seriously, stop worrying about losing your partner, because he will be no real loss. If he thought for two seconds about your situation - your daughter, your grandparents - he'd see where you were at. He'd see that you are making sure you are available to them. But he cares for their welfare as little as he cares for everyone else's. And he's not exactly caring about your welfare either, is he?
So if he's "on the brink of ending our relationship" I'd just save him the time and dump him. The relationship isn't working, you're incompatible. Why prolong something that doesn't suit either of you?