Oh pet [hugs] You've done a great job- raised two kids on a shoestring, kept working and run a household on whatever he deigns to give you. So how can he (or you!) say you can't deal with money? You've been doing it for years. Don't keep running yourself down.
He seems to be the type of person who knows the price of everything and the value of nothing. Because he earns a salary he can attach a monetary "value" to his work, but no-one pays you for cleaning your own home, doing the grocery shopping, washing his clothes, providing childcare, cooking his dinner. You can't work more because you are doing all that to make his life easier and so he doesn't have to pay someone to do those things. Why should you be scraping money together for a coffee with a friend because you've sacrificed your earning potential to facilitate him and his odd working hours? How on earth can he sit by and watch you pay for necessities for the kids on the little you earn and still bitch about how little you bring to the table?
If he wants to play silly buggers I'd tot up all the hours I spent running round after him/the kids every week and multiply it by the minimum wage (£7.20 at present), then ask him if he thinks I still don't contribute to the household, or if it's "fair" that I'm expected to put in the same as him.
DP and I are sorting out finances for our first home together and yes, we're getting a joint account. But as he earns 1/3rd more PA than me it wouldn't be fair to split things down the middle, 50:50. We're each putting in a % of our salary, so that each partner receives a certain level of spends proportional to their earnings, but the higher earner is also contributing more. I don't think that's feasible in a scenario where one is effectively a SAHP/v low earner, and one is ringfencing a large amount of money to spend on a jolly at the expense of a trip all four members of the family would benefit from. I'm so angry on your behalf!