I’d say it’s not clear cut. Both my husband and I have always worked full time. Neither of us teachers. Apart from 1 week together each year for a family break, we never took annual leave at the same time because otherwise holiday cover was too expensive (and in our area, once school age there was virtually nothing with wrap around care so all holiday clubs were 9-3:30 which were useless to us).
If one of us were a teacher, assuming our kids school holidays matched the school that the parent was a teacher at, I’d expect that the teacher would be covering all of the school holidays and the other parent booking time off in those holidays to spend time with the kids too.
You said your DH takes some annual leave in term time, so in that case I think it’s only fair you get an equal number of “you” days that he provides cover for you to do what you want. If he were to only take time off in school holidays to spend time as a full family I’d say it’s not fair of you to then swan off to see friends.
Ultimately, when our kids are young, all annual leave needs to be reserved for school holidays really. I find it selfish to want to use that limited time you could have with the kids when they’re at school, so I could never do that. I would look down on a DH that didn’t want to spend his annual leave with his kids. 20-25 days a year is nothing, and when they’re at school. Time for us parents to see friends etc comes back when they’re get older.
That said, as a teacher you’re getting a lot more one on one time than your husband, so maybe he would/should like a couple of days alone with the kids.
Like I said, not clear cut!