We tend to alternate between being away and being home - and when we're home we switch between having dinner just us, at ILs or with my parents.
If it's a 'home' year, the morning is usually relaxed, just me and DH. We open presents, have breakfast then go to my sisters to see her dc.
Then either go to ILs or my mums for lunch if we're doing that.
Either one tends to be similar. Guests vary - sometimes SIL & BIL and their dc, sometimes just us at ILs.
At my parents it's always the same people.
If we’re home for dinner, again it can be just us, my parents, my sister and her family, both or ILs.
For the evening we try and invite the people we didn't have lunch with to come to ours for cheese 
Last time we were home we ended up with an unexpected houseful which was really nice. We ended up listening to music, chatting, teaching the kids some card games and just having a good catch up.
If no one comes, we chuck a Christmas film on and eat cheese just the 2 of us.
If we're away, Christmas Day is totally random depending on location, but I don't think that helps answer your question OP.
I definitely think changing the guest list is fine, especially if people are ruining it for you. If you do it now, people have plenty of notice to make other arrangements.
Or if you host, can you arrange to visit someone late afternoon so the guests have to go home?