I can see if people are actively complaining about something like that, which they actively chose, why people would question that and find it annoying.
I find some do that, and can be loud about it, while many others are just coming up with a reason when being asked why they do things that others view as different. I imagine some of the latter may become the former if they get fed up with being asked, others I find just avoid the topic.
I can't recall when my parents took their tree down or when it went up, though my mother did a lot of advent and Christmas traditions, largely blurring them all together. I do recall my maternal grandparents, who were very devout Christians, never put up a tree. They had a decorative tree statuette that went up on the stand the TV was in and that was it for decorations - presents were passed out from their chairs. I always thought it was because of my grandmother's disability and preferring peace/clear space - but never really thought to ask, never thought it was my business really. I loved that little statue probably the most in that time.
I do dislike cutting Hallowtide short - it doesn't end until after the evening on the 2nd of November - and would really prefer if Christmas would at least wait until at least after Remembrance Day or even better the start of Advent, but I know it makes some happy to move straight from Halloween to Christmas and as long as they don't expect of it me, more power to them.