@SquirrelsAreNuts I 100% understand why you are upset. That is an awful reaction from him and somehow he needs to understand that. He upset you, and has (or will when they find out) upset the children. Whatever the reason, his behaviour is ingrained and deeply rooted. I don't think you can make any headway on this without external professional support - both emotional/psychological and financial.
Once your (understandable) upset has reduced a little I suggest you make a calm proposal about you both getting counselling support and financial support. Let him know that this has upset you so much that you are considering whether you have a future together, and that if he refuses to participate you will have to consider what other options you have. Research the support options beforehand, talk to potential professionals, even investigate possible dates. This way you are offering him a concrete proposal rather than a vague 'I think we should'.
I wonder whether you have an oversight of what your financial situation is on an income/expenditure basis? Are you eating into 'his' savings to pay for holidays etc? Are you leaving savings intact and just about breaking even on income/expenditure (including enough for holidays etc), or do you have sufficient to pay for all of that AND add to your savings?
My dh had some similar tendancies, although not as extreme. It was born out of an early adult situation of him being completely broke, and being too embarassed to ask for help (for fear of judgement by his more financially secure parents and siblings - and to be fair they would have judged!). To keep a roof over his head, he sometimes had no money for food. One week he survived on a sack of cheap onions.
Once things improved he became an anxious hoarder of money - the more he had in savings, the more he worried about having less than he currently had. 'Eating into your savings' for whatever genuine reason, became anxiety inducing and even though savings were for a rainy day or emergency, he panicked if he actually had to spend them, and wanted to have enough in his current bank account to also pay for unexpected costs. Savings had to be protected.
He is also a bit naive about money, e.g. before we were married and when we had no joint account, his stock answer to any of my discussions about money were ' well I've got savings and you know that those are for us both'. Except they weren't. They were his savings in his account and I had no recourse to them. In that phase of life I was juggling 0% loans, paying back before the term concluded, and things were really tight.
Things changed gradually - we bought a joint house and I insisted on a joint bank account for everything related to the house: mortgage, insurance, food etc. We agreed how much we would each pay into that account (salaries were paid into our own individual accounts). I was the bigger earner so I paid more.
Then we got married, so legally 'his' savings were also mine, although in all practical sense I had no access.
Then things changed again when I benefitted from a small inheritence and used that as a way of suggesting changes.
As a pp suggested, we have different accounts for different things (colour-coded according to the bank logo / bank card colour). White: A sacrosanct savings account for a terrible emergency. Blue: A holiday / eating out / joint recreation / family birthday present etc. account (the pleasure account). Green: Our own individual accounts where we pay for our individual eating out / hobbies / birthday presents for each other. Yellow: A joint account for everything related to day-to-day and annual survival: insurance, utility bills, cars, groceries...
We have now progressed to the stage where all our income goes into the yellow joint account. There are standing order / direct debit withdrawals: to the white savings account (so dh can be reassured that everything is still OK); to the blue holiday account - the balance obviously fluctuates massively but essentially once it is spent, it's spent and then we can't eat out any more or book expensive holidays; and green - we both get equal amounts for individual spend and it our own responsibility to manage within our means. Everything else remains in the yellow joint account.
dh (I think) is reassured that we're not eating into the savings, which allows him to bemore comfortable about spending from other accounts. He knows we're not overspending on holidays or luxuaries. He's in charge of his own account. And as long as the general monthly balance in the joint yellow account remains constant (or increases) then we are living within our means.
All of the above must seem like an overly complicated system, but it does allow me to relax, to not have to negotiate over every spend. However it has been a long-term gradual process.
OP, you do not have the time to achieve this before I suspect you will crumble and get out.