You've been conditioned by him to accept this and none of what you describe re him is at all petty. He is abusive; this is all about power and control.
He is abusing you on many levels; financial, emotional are but two abusive behaviours being shown by him here. He does not give a monkeys for either you or your children.
Unsurprisingly he has refused marriage counselling. Such men do and it would have no effect on his behaviours anyway.
What do you want to teach your children about relationships here; after all they are learning from the two of you. Is this really what you want them to replicate in their own relationships as adults?. They could well go onto do this in their own adult relationships. Your children are being emotionally harmed here by him because they are too clearly seeing the awful ways in which he treats you; the person after all he is supposed to love and cherish. He does not know the meaning of the word.
Read "Why does he do that?" written by Lundy Bancroft and it may well open your eyes. He is controlling and controlling behaviours are abusive ones. He learnt this from somewhere; perhaps his Dad is controlling. Whatever, it is deeply rooted and learnt.
You are a slave put in a cage of his own making.
Re your comment:-
I don't want to leave but what else can I do to fix this?
What do you get out of this relationship now?.
Why do you not want to leave? (that is also a genuine question). Is it due to fear, shame, embarrassment at having "failed" somehow in your marriage. I ask as these thoughts are often behind such comment.
You cannot fix this, there is nothing fixable here. Also he would have to want to put the work in and he clearly does not want to because he has everything that he wants from his life. He is clearly getting what he wants out of this, but what about you?.
Ultimately you have no option but to separate.