Yes, I struggle with this, or something similar. My struggle is with DP's expectation that, because I work PT, I will look after both my DSs and his DSSs in the holidays when I am not working. Our usual routine is worked out around my days off so that I am here to pick up from school etc. I find it stressful, and I become resentful. I have now put my foot down and said I can't/won't do it, at least not on a regular weekly basis, but that has resulted in him being grumpy and accusing me of not treating his DSSs as part of our family.
I understand where he is coming from, but let's face it, it isn't him home all day with 4 kids. It is me. I've also said that it is my/my ex's responsibility to organise childcare for my DCs when I work, and it is his/his ex's responsibility to organise it for his DCs when they work (both full time), and not to rely on me.
Your issue sounds similar. I have tried to explain how I feel to DP, but it always just sounds as though I am being unreasonable. It's not my DSCs that I find difficult per se, they are great kids, it's just the number of them all together, and finding things that they are all happy to do that I find difficult, and also the sorting out fights between them!
In your situation, I imagine your DSCs go on holiday with their mum too? So they are getting double whammy. I don't see why you shouldn't be able to go away with your DCs (presume they are your joint DCs?). My ex goes on holiday with his DW and their DS without my DSs. It has never been questioned by my DCs.
Could you go away with your DCs and your parents? Somewhere in the middle of you both? Or with a friend?