Absolutely this.
It's happened in the changing rooms of my local gym - 'girl dick' swinging around for all to see - and the usual namby pamby excuses about 'inclusivity' from the gym.
So the complainant took her complaint to the national media, and kicked up as big a stink about it as she could. The EA was quoted, together with the proviso in that Act for single SEX spaces where these are necessary.
I still attend that gym. Interestingly enough, they take huge care if a male worker needs to attend that space for any reason: a member of the gym staff will come in and give notice so that women can cover up if necessary, or know there's a male present if they need to come out of the showers or sauna.
If I see this, I, too, will be making as big a noise as humanly possible. Complaints to gym, reassurances that they'll cease with that practice, if no luck there, MP, gym franchise's Head Office, local media, national media, local women's rights support networks.
If they're intent on riding roughshod over women's rights, dignity and safety, I'm intent on making this as difficult for them as I possibly can, within the confines of the actual law (not the law as Stonewall et al would like it to be). And I'll quote it at them until I'm blue in the face if necessary.
OP - kudos to you for challenging the bloke in the first scenario. I would have, too.