DH gets one week off during the summer holidays, when we have my DSC. We have a 2 yo and as of October will have a newborn, plus I have a 6 yo from my previous marriage. My exH is taking me to court for extra contact - previously he's had no holiday contact by his own choice. Now he wants the first three weeks of the summer holidays, when DHs week off always falls (as well exH knows.)
The week DH is off is the only time he can have a holiday with the majority of the DC but obviously that means DD misses out if her father is granted the contact he is asking for. Her father constantly tells her that he's her only and real family and that I love DH and her new siblings more. If we do something when she isn't here(even little things like going swimming) he'll tell her that he would never do something without her because families do things together so we clearly don't see her as a proper part of the family 
I don't want to give him more ammunition to upset DD, but at the same time - she'll have holidays withhim so why should her siblings sit around waiting for her return, and miss out on a holiday with their dad?
Aibu to not want to go without her and cause upset but also unable to see a way round it?