Good god. IF this is genuine... you would be massively better off without him. Reduced bills, less food costs, single person council tax rebate, and child maintenance from him. Probably less cleaning and admin too, and almost certainly improved mental health as you won't have the frustration of him sitting on his arse while you do everything AND pay for everything. It's financial abuse, and if you can't get angry about the effects it's having now, please consider things like your children's financial future (if they want to go to university, for example, will he pay? Or will you end up trying to scratch the money together out of your already massively smaller savings?), and what happens when you both retire and he's got a bulging pension/savings pot while I suspect you'll barely have enough put aside to live on? (I wouldn't have ANY expectations of him sharing his pension pot with you, based on his behaviour so far - most likely this is when he'll do a runner!).
Seems to me you have two choices. One is to demand that you put EVERYTHING into joint accounts which pay all the bills, and from which you each get an equal amount to use for personal spends/savings (check that your pension contributions before putting it in aren't totally unbalanced, too), and that you also divide up all the housework, admin etc. fairly, and failing that you will show him the door.
But possibly the better option is to address the fact that treating you this way says very clearly that he doesn't love you, indeed doesn't give a shit about you and is very happy, nay INSISTS on maintaining the status quo where you pay for and do everything, and has tried to guilt-trip you and called YOU "selfish and money-grabbing" for insisting he gets fractionally closer to paying his own way rather than sponging off you. If you consider that thoroughly, do you really want to keep him, even if he grudgingly agreed to change? Or are you better off getting your ducks in a row (by which I mean protecting yourself financially as far as possible first, so he can't for example empty any joint accounts on his way out the door) and then booting him out and claiming all the rebates and Child Maintenance you're entitled to?