Well, I'm going to go against the grain here. I think he sounds ridiculous. The only reason I can think of, that would excuse his behaviour, is if he's lost a child, and it would be too painful seeing you with yours at Christmas. Has he?
I'm guessing that he's in his 60's or older, given that you have grandkids, and at that age we have ALL lost our parents. You've opened up your life and family to him, and he's basically said "no thanks". I would be extremely hurt and confused by this.
I can't quite believe all the namby pambying responses, about how it would be difficult to have a day with your family. Really? A Christmas meal cooked for him, some drinks, and maybe some games. He's not being asked to work on the roads all afternoon, FGS
Sorry, but he sounds totally lacking in empathy and without resilience. Also quite selfish, letting you go alone, knowing that you won't really enjoy it, as you'll be worrying about how he's doing all alone, crying in to his Bailey's.
Uurg. I bet you've spent most of your adult life pandering to men's wants and needs (haven't we all), and now you're finally getting to your golden years and have to deal with this wet lettuce. I'm a similar age to you Op, and I couldn't be doing with it.
I also noted he's at yours 5 days a week. Is he paying his way?