I can see a difference in someone going out and doing their own thing in a way that doesn't affect their partner, and going out and doing something that impacts on their partner when they get home and for the next day so that they cannot share the load of caring for their children at the weekend, for example.
It is complicated for me, by the OP's worry about her partner straying. My H used to do this and our eventual compromise was that if he wanted to drink, he should arrange to stay with his drinking friends or anywhere he liked, so long as I didn't have to deal with it, and only return when sober.
I didn't like that he still did it, but it didn't impact on me in the same way. It made me anxious waiting for him to come home and I didn't like it when he came home that way. I would lie awake, listening to him breathe, feeling that if he died it would be somehow my responsibility. I also disliked the fact that the children might see him like that.
Once he was 'allowed' to do it, he went out less and less frequently, then eventually landed in hospital one night and could no longer ignore what I had been telling him for a long time: that he was putting his own life at risk and risked leaving the children without a father over having some 'fun' / 'freedom' which on most occasions, he couldn't remember anyway.
Oddly enough, since then, he seems mostly able to control his intake, and also will listen when I say 'please stop'. So he was obviously choosing that pathway. I believe he saw me as an extension of the mother who had nagged him when he was a student and he was still rebelling.
You need to find a way of breaking the cycle that works for you both. He needs to work with you to find a solution that will make his going out bearable, so he can enjoy himself and you don't feel the need to nag. It doesn't sound to me as if this is all about your anxiety. He is being selfish as well.