Just be honest, OP! We go to Norway and Christmas Eve is the big one there. MIL and DH siblings come on Christmas Day afternoon, because they know we do British Christmas as well as Norwegian.
BUT, the first few years we kept getting invited to parties or loads of other friends and extended family wanting to drop by. We just said "no, we like a quiet day I'm afraid. You're more than welcome to drop in through the week/we'll come to you!" - but Christmas Day is small and quiet for just immediate family. They accepted it graciously and now we just make sure there's other days/NYE.
If you make excuses then they'll keep asking. Just be honest about it.