I wouldn't be impressed if I were left looking after the tiddler whilst DH and the DSS went off gallivanting, that's for sure.
I'm assuming still that these are 'proper' going away abroad somewhere holidays?
We had to organise our whole holiday be it away or at home to suit our five. Put it this way, when DD2 was 2 the older ones were 12,11,5 and 4. Now that's an interesting age range to keep occupied and happy simultaneously 
We tend to do beach holidays, where the older ones can do sea and surfing whilst the smaller ones do paddling, seashells and sandcastles. Buckets, spades, bat & ball, mini-boards, towels and a cool box-all very 1970's but great memories. DH and I are able to still be 'together' but it keeps everyone happy. How many children don't want sea, sand, warm squash and either a pastie or cone of salty chips on the beach? We recently had two weeks caravanning I hated the idea of being cramped and insisted on a static with the biters and all we really did was hit the beach followed by BBQ and a film with the 14yr olds (the little biters were usually fast asleep by 7pm courtesy of sea air and exercise). It was actually a fantastic fortnight, with all the biters having a wail of a time despite the distinct lack of electrical games and flashy funfair type stuff.
Other than that had you thought of perhaps doing a local holiday? Looking up the activities available reasonably locally and heading out to a different one each day? Even just tramping through the woods bug hunting is still fun when you're a teenager-especially if it's followed up by a treat tea. All you can eat Chinese buffet is always a winner in our house-the bigger biters get to eat like there's no tomorrow (and try lots of new things) and I've never met a 2yr old who didn't relish eating noodles with their fingers. You'd get to do it without the looks that a family of 6 arriving to eat attract too 
The same principle applies if you're abroad too-I genuinely don't see any need for you all to separate in order to accommodate DSS.
I really think that you ought to have a conversation with DH and express how this is making you feel. If you don't actively change something you are essentially just accepting the lonley status quo in exchange for the promise of an annual holiday with just you, DH and DD and that doesn't solve your problem as regards him leaving you to it on the other vacations iyswim.