I’ve said Yabu but I’m making some (probably wild) assumptions.
First I’m assuming that you are exaggerating a bit when you say you “spend so little time together as a family”. However I guess in reality, it’s true that you may not spend time together at weekends and evenings, that you don’t have summer holidays/staycations together. if he’s one of those husbands out drinking every week with mates after work and playing golf or going to football every weekend, while he leaves you stuck indoors with two kids to wrangle, then sure. He is being very unreasonable to suggest leaving you behind.
Second I’m assuming you will have opportunities to go skiing in future as a family (why wouldn’t you?).
Third - a week of tim devoted to your youngest sounds amazing to me! I would go and stay with my own family and enjoy the heck out of the opportunity not to have both kids to wrangle.
Fourth - my dh has taken my dd on vacation without me to visit in-laws overseas and they had a blast (she was 6). It’s great that he wants to step up as parent isn’t it? I am assuming they aren’t sticking the 3/4 yo in crèche all the time!
I have taken a 1 year old to a ski resort with in-laws and it was okay, but not great. Horrible, horrible long journey. Too young to go up the gondola ski lift due to altitude, not walking well enough to really enjoy the snow, but walking well enough to cause trouble in the totally inappropriate valet accommodation which wasn’t child-safe, didn’t have a stair gate or a high chair or a proper cot etc. It was a waste of effort and money really, as someone had to mind the baby the whole time and we had to lug a lot of baby stuff with us. It was assumed that I would be the one to sacrifice my pleasure and comfort so the rest of family could enjoy their Christmas vacation to te max, as I was the baby's primary carer. I was I told I “had to go” as in-laws expected me to be there so they could “spend time with the baby” when they weren’t skiing.
So I probably wouldn’t be upset but I’d try and get all these thoughts out on the table - what is dh assuming about this ski trip and future trips? What about other holidays? Then I’d bargain hard for something to compensate you. Do you get to pick your summer vacation and what happens next Christmas ?