Hmm, I wouldn't be impressed with my DH calling me selfish over something like that and shutting down the discussion. And if it was me, then I would be trying to persuade DH that we should go. We often go away for Christmas and luckily haven't faced something like this at that time. But we have had to make decisions about going on holiday before (we decided to stay at home once, and to go once - the journey was actually pretty bad, but once we arrived DC was fine, fortunately).
Any time this came up, any decision on staying or going would have to be a joint decision, properly discussed, (and not one party just calling the other selfish!) And if we made what turned out to be the wrong decision with hindsight, then we would avoid any recriminations. I do actually think we made the right call each time, but the point is, even if you made the wrong one, you need to both be on board with it, rather than blaming the other person.
If you actually decide you can't go, and your parents are coming over on Boxing Day, I might be tempted to pretend to the three year old that actually Boxing Day is Christmas Day - unless the other child is older and would realise? That way your DS gets an extra day to recover, still gets to see family on "Christmas" day, you don't have to buy loads of new presents as your parents can bring them.
You can have real Christmas Day as a quiet day just you.