This is going to sound really corny, but I liked the fact we had all that time together in the school holidays. I mean, there is a whole lot to moan about, being a teacher, but that really is a perk (and no, I didn't go into teaching with that in mind either, I had been teaching a long time before having dc).
However, 5 weeks of dh's AL used to get eaten up quite quickly. There were 5 INSET days to cover each year, for a start - that were obviously his by default. Then I liked the fact he came on holiday with us in the Summer, and came away to stay with extended family during some 1/2 term breaks and Easter, and took a good few days off over Christmas to all relax together. He'd usually try and book a day off to go on a big day out somewhere in holidays with us too.
I'd rather have him with me on some of those times than have him taking AL in the school holidays and then not spend it together.
DH would also sometimes take 3 days off near the end of his AL year and do something like decorate a room whilst the dc were at school.
If I wanted to spend a couple of nights away, I'd tend to do them at a weekend when dh was home anyway.
Of course, the difference here is how your dh spends his AL.
So, as a general rule, then I'd say yes, the school holidays fall to the teacher in the relationship by default, but, in your case I do think you need to have a conversation about fairness and how time off is used.