I would not be chilling... Until I read the bit about poor health, and that would totally change things for me.
My ex dp could not be bothered to drive 45miles and spend our first Christmas together, even though he doesn't really like it and it really made no difference to him if he was with his family or me, and so, as an absolute Christmas fanatic, to whom it made an enormous difference, that was a massive nail in the coffin so to speak.
I wouldn't let him spend our second Christmas with me, because all I could do was think about how shitty he'd made me feel the year before and I didn't want a second Christmas ruined.
Sounds daft to a lot of people, I know, but I get where you're coming from OP, if it's a really big deal to you, it's not an easy thing to forget. And you get an annual reminder lol.
The second year, after seeing you being upset that you weren't together for your first Christmas, he should be with you. But, then it's likely to be his last Christmas with his mum, and that has to be his priority. Something like that takes precedent every time. It just has to.
Talk to him. You say you have personal reasons why Christmas is so important to you, and whilst his mum needs to come first, it shouldn't also mean you should be left out... Especially if he's aware of the reasons you find Christmas difficult.
Would you going with him be out of the question? Is it that different to arrive in Christmas day than boxing day if you're going to make a 200 mile trip?
Sending Christmas hugs, I really feel for you xx