Why would that happen? It only becomes an issue if those spaces are open to biological males. One would naturally assume that since men are not allowed, it's simply an androgynous looking woman, so nobody would want to chuck them out.
At one time I worked in security. I was on guard at a music festival. The ladies toilet had a long line, so one drunken woman went into the men's room. This was before being trans was something that was widely talked about.
She was not androgynous at all. The men in there heard her sit down to pee. One man was outraged because his young son had seen her bare ass as she was too drunk to close the door properly. Other men were upset because she had seen their dicks at the urinals. I then had to throw her out of the show because after I warned her, she tried to do it again, then became belligerent when challenged. Her drunken excuse was she was actually a man.🙄
If I did the same thing now, I could have been accused of "bigotry" for my actions, even though I did it to protect the privacy of those men.
The point of telling this story is that who is going to assume, without clear evidence, that anyone is the opposite sex if the spaces are kept single sex? They would need a good reason to think so, which in that case the men had. I highly doubt they'd have complained to me about an androgynous looking person using the men's room.