His message was manipulation disguised as 'reason'. Bottom line is he used the threat of 'what will the children think of YOU if you don't have me back' conveniently forgetting that it was HIS emotional abuse that caused the split in the first place! He's trying to imply that somehow the cancer trumps his behaviour. It certainly does not.
And the children aren't going to ask 'why you didn't come together before the end'. It most likely won't even occur to them that you should. And if, by some remote chance they do ask, you will simply tell them an age appropriate truth. If you choose not to share that their father was emotionally abusive, you tell them that sometimes married people can no longer live together and that when that happens it's better that they live apart.
It's an insidious thing with emotional abuse. If he'd had the 'habit' of beating the shit out of you I have no doubt that you'd find it very easy to say "Oh hell no!!" to him moving back. Because the memory and the fear of physical pain stay with us. But with emotional abuse, as we no longer live with it day to day, sometimes the edges 'blur' and the hurtful and manipulative things begin to lose their 'power' as we get further away from it. This is what makes it so hard to, as you say, 'maintain your line'.
If you think it would help, remember and write down past (and obviously, ongoing) instances of his abuse. Fold it up and stick it in a drawer. And when you weaken, take it out, read it, and tell yourself "that's why we aren't together anymore".
He doesn't have to be living there for you to be 'safely supportive' from a distance and to do what is right for the children as far as him spending time with them. But your role is no longer to be his primary carer. He forfeited that right with his behaviour. You do first off what is right for the children, secondly what is right for you, and thirdly what feels 'safe' for you to do for him without sacrificing your 'self' and your happiness.