If he isn't 'into' Xmas then why does he care where you spend it? Would he feel the same way if you took the same amount of time off at a different time of year every year to spend with your parents?
But another question is what exactly is 'not into it'? Some people enjoy Xmas but want to spend the time quietly, minimal fuss, just being together. Just because he doesn't want a big hoo-ha doesn't mean the Day itself isn't special to him.
Is there room for compromise? Being gone a shorter time period or perhaps alternate years spending the actual day here vs there, one year with your parents on the 25th, the next year going before or after? I have a feeling that what your parents enjoy is having YOU there, as opposed to having you there on the 25th. TBH, spending an extended period of time in a smoker's house wouldn't be something I'd do, either, but I'd try to find a compromise.
But in a larger sense, this has nothing to do with Christmas. This has to do with partners issuing ultimatums. I agree with ultimatums when it comes to addiction, abuse, hoarding, etc, real 'deal breaker' stuff. But not when it comes to something like where to spend Xmas or where to go on holiday. These are things that can, and should, be compromised on. If your partner won't work on a compromise, then you need to reconsider the relationship.