wow, he sounds like an absolute darling!
in my opinion, and experience, the very thing you are being accused off, is what they are doing themselves, and the reason you are being accused is because if they can do it, so can you!
Turn this around on him, ask him why he is never around with the kids, why is he not doing family stuff on the weekend.
I'm really sorry OP but if you want this to stop, put your foot down and tell him you are going, do not ask his permission, he is not your boss
Myself and DP are far from perfect, but when it comes to nights out etc, as long as we can afford, got nothing going on as a family, then its all systems go. i will often ring him in work saying been invited out on such and such date, can you think of reason why i can't go....its not that i'm asking his permission, merely giving him the courtesy as my partner to let him know i'm going, which works both ways xx