As pp has asked, delicately, how terminal is he - are you talking weeks, months, years?
Will he be having carers? Can his family support him or can the visit/support him if he does come back? Has he got lots of treatment planned? How is he expecting to get to/from appointments?
The kids are going to be traumatised enough about losing their dad so young. Add in them seeing him poorly and potentially feeling like you've refused to let him come home (they're young but it only takes one little seed of doubt whether planted maliciously or otherwise) then I don't envy that minefield.
The biggie for me is the emotional abuse. Again, short term, and on your terms, maybe he could see out his last weeks. Anything longer no way. I bet its taken you a livetime of strength to get to where you are now. You see posters on many threads being told its OK to step away, or refuse to help care for elderly relatives when their own mental health is suffering, so I'd support you to refuse to help in your case op.
Whose house is it - have you separated all finances yet?
If he had a few months with a full care plan in place and respite and hospice available for when it's needed then I'd be thinking about it.
Longer term, then no.
He has a family to fall back on. You can still support him seeing the children and facilitate contact for the foreseeable. He is not your responsibility though op.