OP, he's mad. 🙄
You both work full time and you have paid off the mortgage, so the most expensive bill each month doesn't exist for your family. Even if all you both earned is minimum wage, you have enough to live on and can afford a normal level of treats. Things like kids activities, coffee out with friends, a trip to the beach. As well as basics like new clothes when needed, suitable shoes, dentistry and pharmacy meds. He's a tight arse.
He's got life how he likes it, it's not a hardship for him to live this way. That he doesn't care if it's a hardship for the rest of you means he either doesn't care about you, or doesn't recognize you as separate people who aren't a part of him and who have their own needs and wants. Either way it's not good.
All he sees is people spending money and he doesn't want anyone else to have any fun because he wants to keep all the money. Who made him god of the household budget and all decisions?
It doesn't matter what others think of him, they don't have to live with him and his meanness, they don't see that side of him.
What's this shit about "it's not his job" to collect DC from friends if necessary? It's as much his job as it is yours. Don't feel guilty for going out sometimes, it's a normal thing to do.
If he hates literally all people he shouldn't be in a family then, he should be single. Then he'd have to pay rent/mortgage again probably as well as child maintenance, so life would be more expensive for him!
Although TBH I wouldn't put up with someone wanting to stay married to me if "it's cheaper" was the only reason! You're not there to facilitate his chosen way of life. It's ok to live the life you want. If you'd be happier single/looking for someone more compatible then do that. It's upto you.
If his problems are down to anxiety then he needs to recognise that it's his problem and he doesn't get to relieve his anxiety by controlling everyone else. He needs to find solutions that involves him doing things to combat his anxiety, not others being expected to relieve it for him by changing their behaviour to the extent they no longer have any kind of normal life.
There's every possibility he's just a dickhead and it's nothing to do with anxiety though. I'm sick of all bad behaviour being blamed on MH issues. It's entirely possible to have MH issues and not be a dickhead. Just as it's possible to be a dickhead as a personality defect with no underlying cause.