I think the reason you're getting a lot of stick is because the idea of time to ourselves in the 'holidays' is a luxury most of us only dream of. And that doesn't mean it's wrong to sometimes have those days I'd you can make it work. Your faux naevity smug about 'but that would make my weekends busy with admin' stings when that's the reality for most of us, and many of us are doing very full-on jobs with very little time off.
From the sound of it, you've either returned to teaching after a break, or because children were in childcare, you had much of the holidays off. Sounds like this is your first experience of life either being work or looking after children, without a break.
It might not feel it, but you guys are in a privileged position because between you, you have more leave than school holidays. So unlike most families, you can take time for yourselves. If he has 25 days, then perhaps earmark 10 for holidays or time together, 5 to cover sickness, 4 to cover school events, and then 3 personal days each.
Personally, I'm self-employed, and take most of the holidays with the kids and then catch up with work in the evening - a total double shift.