I think you're right, he's not being entirely fair.
Unless you did something heartstoppingly exciting with the kids when he went ski-ing??
If you just hung (hanged?) around at home then I think you should say to him 'that's a lovely idea but i wanted us to do the really special things together as a family'. I would try and think of an alternative, something nice but not so spectacular - could he take them to a city with great childrens attractions?