Because a Saturday is not a working day for either of you - it is very different from using up precious, limited leave! You are in a career that is part time (over the course of the year, with your unpaid leave). He is full time and therefore has much less leave.
The first year of having a child in school often plays havoc with leave allowance, as many bank it up expecting a slew of viruses, bugs and school events. If that has not occurred, great! But it does mean you may get to the end of the leave year with a glut that you cannot carry over.
Honestly, most of us struggle by with a patchwork of leave, calling favours from friends, holiday clubs and (if available) family. The concept of deserving days to yourself whilst on leave is absolutely alien to most working parents. DH and I are often working through our holidays too, nature of our career, which doubles the stress.
DM taught and looked after us each holiday. DF joined us on family holidays, and used the rest of his leave for covering Baker Days, illnesses, school events, dentists etc. that DM couldn't cover due to her job.
Talk tp your DH calmly about allocation of leave for the following year, like other parents do. Take into account your lack of flexibility during term time and leave him with a buffer. Use weekends where possible for small events - I would be pretty hacked off if my DH demanded I used a day's leave so he could go for a haircut and a coffee with his mates without the kids in tow.