These arguments get dafter and dafter 🙄 A very tiny proportion of men can pass as women in person. If such a woman-impersonator goes in the Ladies, behaves as women behave, washes his hands and leaves, why should I care? That level of deception takes commitment. There's never a guarantee about any human activity, but I'd say he's fairly unlikely to blow his cover by even speaking, let alone molesting anyone.
Those few are not the men demanding all-areas access to women. As you say, they can stealth it. They don't need to demand or force it. They'd lose out by doing so. I don't care much about them.
Back in the day, transsexuals would occasionally turn up in the Ladies and they were trying hard to just blend in, keep their head down. They were identifiable as male but, back then, they really did want to 'be women' and there weren't vary many of them. We were kind, mostly, because we could afford to be. We just left them alone, they didn't seek attention.
Those few are the men losing out to the ones demanding all-areas access to women. They are recognisably male, like the men seeking access for darker purposes. I'm sympathetic to them, but I'm also sorry we were permissive because that is the inch we gave; activists have taken a mile, and then some. I don't care enough about them to risk the privacy, dignity and safety of all women and girls.