YANBU it should have been discussed/ raised in private but its out there now so it needs discussion.
Are there more issues here than the going back to work statement?
Is he under a lot of pressure and in need of a break that maybe he feels he can't have because of finances, maybe he feels you don't need because he hasn't run a house of 4 children + activities?
Is he going through a mini mid life crisis missing the dual income lots of freedom pre child existence that you had?
If money isn't really tight/ you're surviving could it be that he isn't feeling appreciated as a man/ lover and he wants you to be more than a mummy?
Could you write a list of all the activities you do and manage and review which would have to change if you worked school hours only. For example I should imagine that 4 hours of work a day, probably minimum wage type stuff even if you could find it, would be negated by the school holiday club costs for four. Evening work would mean he does all the post school runs and club drop off/ pick ups. This would show some willing as a start to opening discussion and might enable him to open up with where this has come from.
How much extra funding would make a difference? Would £2000 a year pay for a nice holiday and that's the thing he craves or is it money for having a hobby as an adult himself, would a monthly adults night out with a babysitter at home make him feel more content with the situation?
I'm a stay at home mum (now) haven't quite got it right yet my eldest is 7 and autistic so quite a handful and clubs/ holiday placements aren't an option which is one of the major reasons I'm at home, DS2 is 4 and DC3 is due any day. I've done and redone the sums for our situation and find that ebay is my best option for earning a few thousand a year. We sort out our unplayed with toys, clothes, diy stuff, stuff from around the house and general excess stuff, I ebay for my mum and family for 50% of the money made.
There are some other savings I find by being at home I shop around to buy stuff we make considerable savings. My shopping bill for the family is about half what it was when I worked full time and we eat better.
I don't know about your situation and don't wish to project too much of my own onto yours but I do think its easy to become just a mummy (I'm not dismissing that it is a very important job) and so much involved with the children and their lives that you forget/ don't make time for being a women and lover yourself.