If you'd said he pissed on the stairs, brought drugs home and generally never helps with the kids on normal days aka is a shit dad who has opted out of all his parental responsibilities then you would have gotten very different responses.
You keep saying you can't believe people think it's acceptable but if you'd included all that important information in your OP then most people wouldn't have called you controlling, they just would have said he is clearly a shit dad and shit partner in general and being with him is an unhealthy choice.
For you and for your kids.
You say other people who like nights out aren't putting their kids first but unfortunately making the choice for your kids to grow up seeing their mum do all of the cooking, cleaning, childcare and bearing all the mental load is not putting them first long term.
You're teaching them that women live to make other peoples lives run smoothly and men live to do what they want. That a mum should put everyone else first but a dad should always put himself first.
These nights out aren't the worst thing about your relationship.
Having two kids in succession with someone who was a prick after the first one means that presumably neither was planned. Of course that happens sometimes and it's done now.
But I think doubling up on contraception from now on is very important. Unless the second one was planned which would have been irresponsible on both your parts knowing how little he's stepped up with the first one.
Do you really want to spend the rest of your life with him 'for the kids' even though it's teaching them that this is what a relationship looks like and increases the likelihood of them ending up in a similar one as adults?