I think it is telling your DS has spoken to you about this. I suspect that neither of your DC want to be spending any overnights there, never mind Christmas
I know you want to enable their relationship with their DD, and not want to appear confrontational or stopping contact. Which is admirable. If your XH was being the father you want(ed) him to be.
Unfortunately this is not the case(and hasn't been for some time). He has moved from being neglectful and selfish -not cooking meals, taking them to stand around outside when ill, not giving them the time they need, forcing unwanted relationships with OW/her children, and no doubt more- yo verging on/actual abuse.
I know that you have said you'll come and get them. But what way is that to live? They are anxious at best, you are waiting on a call to come and rescue them from a possible situation involving alcohol, drugs and violence. I know myself sometimes we have to make difficult decisions to protect our DC. I am not suggesting no contact at this point. I am suggesting no overnights, and Christmas with you. I expect your DC will be relieved (even if they don't say so). And it just might waken up your X to what the consequences might be in terms of his future relationship with his DC. I also know you will be concerned about him simply using it as an excuse for walking away. But, if he does that is what it will be. An excuse. If he does, well that is who he is and your Dc are better off knowing now rather than later.
Worth logging it with a lawyer if you think it is necessary. Better the consequences of you standing up for them than their distress if you don't.