I think the key thing in your particular situation is that your DSD only gets to see her dad 52 days a year, whereas his other children see him every day.
He should be using his annual leave to see as much of his children, ALL of them. as possible.
Assuming he gets four weeks off a year, that's 20 days. If you add that to the 52 days a year he currently sees his first born, that's still only 72 days with 1 child, compared to 365 that he sees your joint children.
If you go away for a fortnight and don't take DSD, he'd be seeing her for 62 days or less, so spending 300+ days more a year with his other kids.
It the time to build that parent-child bond that's the issue here, in my opinion, not the holiday itself.
Plus, surely your DH is gagging to spend more time with his child that he barely sees. Does he not want her to be on holiday with you all?