Providing you can prove (hard copies of emails etc) that you have offered your ex a number of options I would suggest that, if you haven't done so already, you go ahead and book your holiday for the final two weeks of the school summer break.
Then write again to your ex and say that, in the light of his failure to confirm what contact he wishes to have with his dc(s) during the school summer holidays, you've had no alternative but to book your own holiday with dc(s) from/to (give dates) otherwise any further delay could have led to him/her/them losing out on an away break with you.
Also give your ex a further choice of dates* during the first 4 weeks of school summer hols so that he can choose a 3 week period for contact with his dc(s).
*I suggest you allow a couple of days after end of term for your dc(s) to recover from the ardours of school, and a couple of days the other end for packing for your holiday. This will leave a 3 1/2 week period of available dates that you can put on the table for your ex to choose from.
On the basis of what you have said in your OP, should your ex refer this matter to his/a solicitor I doubt that he will be encouraged to make an application for further direction or variance of an existing Order relating to contact but, in any event, a Court is unlikely to look favourably on any such request as your evidence will be compelling in that you have reasonably attempted to reach an accomodation with your ex by offering a choice of dates to comply with existing contact requirements.
In addition, you are bound by circumstance in that your dh is restricted in the length of time, and when, he can be absent from work while, as a teacher, your ex has an extended holiday period during the summer months.
I suspect that your ex is simply trying to be ornery but that's no reason why you should roll over for him. If you want to throw in a sop to his ego, you could perhaps suggest that this year he has the dc(s) for the first 3 weeks of the hols, and next year he can have them for the last 3 weeks (if this suits your dc(s) and yourself).