This. All the 'well it's a religious holiday' stuff is nonsense.
The tree, the lights, the food, the evergreens, mistletoe and all that, all pagan. Pretty much all cultures (that have dark-light seasons) have a mid-winter celebration to get you through the dark months. It's really unlikely Jesus was born around the midwinter solstice when the cultures already existing were celebrating their festivals with food, lights and evergreens. Ditto Easter, April 1, New Year, Harvest festivals of all sorts, all plonked onto pagan festivals.
I suppose if the Christians want to keep presents, the star on the tree, carols and praying, oh and do it in April, the rest of us will keep December 25th, trees, lights, food, evergreens and all that. Including Terry's chocolate oranges because they are CLEARLY heathen.
As for your DH OP, it won't get better. He doesn't respect your difference as much as he thinks. And the children will choose at some point and that will be an issue.