We do it, have done for the last 3 years now. We used to alternate Christmas dinner with my family/OH's family, but we would always have to still visit both sets of family on the same day. As you can imagine we spent the majority of the day carting things from one place to another & used to end up getting home at 9 or 10pm, knackered & not being able to do much together as a family on our own on Christmas Day.
So 3 years ago I suggested we have our Christmas dinner on Christmas Eve, just us, in our own home. We spend the day playing board games, listening to xmas music while cooking dinner & then after dinner we go to our local pub for an hour to see some friends (family friendly pub). Then we get home, open a few christmas eve presents (new pj's, a mug for hot chocolate & a book).
Then Christmas Day we still visit both sets of family but we don't have dinner with anybody, so we work our visits around what time they eat. My family do breakfast on xmas day (bacon sandwiches, sausage sandwiches etc) so they keep some for us to eat when we get there. They eat xmas dinner at 1pm so we leave just before they start serving it up.
OH's family have their xmas dinner at 6pm so we go to their house straight from my family's. They put a few nibbles out in the afternoon & we leave before they sit down for dinner.
It's been great. We're home by 6/7pm so get to enjoy a few hours together in our own home. We have cold meat etc for our evening meal & it's just so much more relaxed.