Ok - I am not in favour of holding passports as that is controlling - but you are clearly in breach by not sharing details, and by picking dates that inpinge on his contact. You are being deeply unreasonable, more so when you refuse to swap the weekends around as its "your weekend" well, it would seem that he is doing this right back. He is well within his rights to insist on his weekends of contact and if that ruins your trip - well thats your fault for not doing it.
All that being so.... i heard a thing on the radio yesterday and it was a convoluted case of a woman who had 7 babies taken into care and stated that she was going to have more until they let her keep one. The social worker simply said "where are the interests of the child in this decision".
So what would suit your daughter? I think offer to swap the weekends, fess up to the mistake - try saying "I was wrong" and offer a concrete alternative to the time - ie a few more weekends at his choosing.
She will get 3 sets of benefits - more quality time with both parents, an uninterrupted holiday and finally - less stress and anxiety from you both.
Offer the hand of friendship and co-operation, if it is batted away - offer it again and again.
The court order is a minimum framework - you have the ability to agree things between yourselves - if you want him to give ground - so should you, offer it freely and openly.