I think YWBU to refuse to have them when he goes on holiday - just as he is BU to refuse to have them accommodate a break for you.
Agree with PP's, next time he asks you to have them extra tell him the dates you would like in return (before the dates he wants, so he can't back out at the last minute).
Ultimately you can't stop him from going on holiday without his children, and as their other parent it is your responsibility to look after the children if he does that. But it works both ways and it should therefore be an expectation that if you want to go away or have a break, he is expected to look after them, no questions asked!
For what it's worth, DH and I holiday together twice a year - a week away in the UK during school holidays where we take DSD with us, and a week abroad during term time where DSD stays at her mums. Contact is 50/50 . If we've been away without her, we always arrange to have her extra as soon as we are back so that her mum gets a break too, and vice versa - her mum always has her extra when we have taken her away for a holiday so that she can get some time with her to catch up. Whenever her dad or her mum need to change contact, the other is always accommodating and both use he opportunity to balance it out as well. E.g. her mum wants her on 'our time' to take her to a family event, well that's fine as we will have a family event coming up in a few months that falls on her mum's weekend with her, so we sort it out then.
Your ex is being a dick by refusing to help you out when you've been so accommodating to him in the past.