We had adults visiting the house on Christmas morning (neighbours and work colleagues of DPs), but otherwise it was just our nuclear family (2 parents and 6 DCs). I know DH's experience is different as there are wider family close by who visit after the lunch.
Now that we are a family ourselves, we have done some Christmases where we visited both families (only 20 minutes apart, while we live 2.5 hours away), and some where we stayed at home (even before DD arrived). When we visit, we almost always rent a cottage to have a "home base" to retreat to and host a gathering ourselves.
The years we stay at home, we go to Mass in the morning, have a few duty visits to make (mostly adults in all houses due to timing) and then get home. We get the turkey going, light the fire, cook up some nibbles to keep us going, and settle in to opening presents and relaxing.
After dinner, we tend to have some sort of game as a family.
But as our lives are just soooo unbelievably hectic, it is great to get a day where we can actually relax completely. We've played charades, cards, board games, watched particular movies we want to together etc. We all work together on the meal (DD has always been a part of that, in between playing), and actually take our time to sit and chat over the table.
Don't buy into the hype of a large happy gathering as what everyone has. The years we have gone home, we've often had huge stresses involved, usually caused by others - including screaming matches between adults who have all drank far too much or phonecalls dropping bombs. Yes, we've had happy ones too - but you never can tell in advance which it will be. There is rarely a chance for games and the like in either house. There is often an atmosphere in one or both which is not particularly festive.
I am looking forward to a year when I can close a door on the world on Christmas Eve (open it again to go to mass (probably Midnight Mass) and then come straight home again) to a warm fire, nice food, a nice glass of wine and a good book while having some good music (seasonal but choral, orchestral and country rather than "Festive Favourites" cheesiness from the radio). Maybe, when I am widowed and in my 80s I will get that....
(Yes, that last is slightly tongue in cheek - but also a very real desire for peace at some point, whether Christmas or some other time).