I'm interested to hear that men will enter women's spaces anyway, because as a younger woman I was often followed and harassed in public places...and they always fucked off once I got into a women's bathroom. They COULD have just walked in, but for some reason they never did, and in fact lost interest and went off.
Societal expectation is a thing. It's powerful. It's often the only protection we have.
And it's the very negation of this expectation that is, in some cases, causing trouble to gender non-conforming women. Because we used to be able to trust that a butch woman was a butch woman. We could even trust that the very very small number of transwomen in those rooms were also genuine, as the societal expectation was that they would not be there unless they had gone through a process (bearing in mind that you can't change sex).
Now thar toilets have become an opportunity for land grabbing and boundary breakdown in order to destroy women's spaces more widely, we can't trust this any more. And naturally, women are being blamed for it. Not the activists who rejected third spaces and fought to destroy safeguards, boundaries and consent.
Societal expectation is real and powerful. We never relied on anti-man force fields.