You will get the tired old "no attacker is deterred by a sign on the door" argument
and the "no women's toilet is a 100% safe space" argument
and those statements are true BUT if a space is designated for women only, everyone knows a man should not be there. Including those men themselves. And if he's there, he's up to no good (unless he's an authorised cleaner, etc).
Men really don't, generally, want to go in women's loos. The nice men, the horrible men, the drunk men, the desperate-for-a-pee men, the young men, the old men. They just don't.
So why does a small minority of men feel entitled to?