As a parent of a DD with HF ASD and ADHD, (only 12 yet but...), my take on this is:
You have already been away without him, so you know he can cope.
You know he is invited and will make sure to ask him.
You know he doesn't like Christmas and just wants a quiet day.
But you also would like a chance to enjoy a "proper" Christmas for a change.
The clincher part of it for me is the first point above - you've gone away before and he's fine. You are absolutely NOT abandoning your DS. You are putting his interests before your own.
I think that, for a change, this is one time where you actually CAN put your interests first. He is invited, but won't want to go, and can manage fine himself.
So go, let your hair down and enjoy the Christmas that you want to enjoy.
OK, yes, definitely make sure DS is set up beforehand with food he will like (HE will like - that may have nothing to do with festive treats and alll to do with beige food or microwave noodles, or just a fridge full of cheese) and options of things to do. Leave him a Christmas present if he would like that. And the usual, emergency contacts (which probably includes the local favourite takeaway that he can get to or delivers), talked through the routines and "what to do if..." stuff.... And probably some emergency cash as well.
Have a wonderful time.