I think you were being more unreasonable, tbh. Surely the whole point of using cultural references is to refer to a shared culture? If you were American and he British, and he pretended he didn't understand then he would be being unnecessarily rude, but as, presumably, neither of you are American then using the cultural references of a culture neither of you belong to is, frankly, odd!
Particularly with the first example, I also disagree that you would be able to make sense of it out of the context.
If you had used one of the examples above, e.g. 'Well I'm not going to invite them for Christmas,' that would mean, as you were trying to communicate to your DH, the annoying people would have been disadvantaged by their behaviour because you would not invite them to a significant social occasion. But what you actually said was 'Well I'm not going to invite them to [an event I would never have invited them to anyway because I don't celebrate it], which makes no sense.
Rather than the implication being 'due to their behaviour, they've lost out,' it becomes 'their behaviour has had no adverse consequences for them,' iyswim? So really what you actually said is almost the complete opposite of the point you were trying to make.
Also, using excessive Americanisms is just annoying! I have a friend who frequently refers to the 'jocks,' at our old school. We lived in Wales.
I just say 'Do you mean the boys that used to be on the rugby team?'