OP, something is very wrong here. I've read this thread with my mouth wide-open. You are being controlled, manipulated and bullied. And you seem to be frightened of this man because you are going along with his controlling, by doing what he wants you to do. I wonder what would happen if he came home and the house wasn't immaculate? Would he be spiteful and unkind and make you feel as if you had caused the problem? Yes, thought so. He's what we call a "wanker". What lesson is this giving your children that he calls the shots and you jump?
Let him come home to an untidy house and see what happens. My husband would never DARE to make such a comment to me. We both try and keep the house under some kind of control, but sometimes that doesn't work - it's just the reality of family life. We share out the meal-making and the household chores because they are his children, he eats the food and he lives here. And I can't see that he has paid for a live-in 24 hour house keeper so he has to do his bit. Sounds like your DH is not pulling his weight as either husband or father.
My neighbour is now going through a divorce after 20 years of marriage. In the early years her not-so DH was just like yours: trips away with boys, late home every night, expected the place to be ship-shape, left her with four young children, sulked if she did anything for herself, especially when she had one weekend away with her ante-natal mum friends etc. 20 years on she has had a nervous breakdown but has come out of that, confronted her marriage, decided to move forward without him and is now getting divorced. She stopped washing his shirts, tidying his house, being 100% on the childcare a few months ago. It has been hysterical watching his reaction. She is a different, and much happier woman and wishes she had done it years ago.
Please get a grip of the situation now, sit down with him and talk about the chores and the money and how you are doing to divvy things up so that life is fair for you both, especially if you are both working.