Really interesting thread.
IMO the main problem when generally discussing sex segregated spaces is that there is hardly ever any analysis of male violence in the discussion and there are never any thoughts given to having a case by case decision made rather than a blanket approach.
It's very different to desire a single sex space to feel free from potential harm, in a refuge or rape crisis service for example, than it is to insist on separate toilets because toilets can be designed in all sorts of ways so they are still safe to use if gender neutral, but equally they can be designed in ways that have not even acknowledged the possibility of male violence and so can never become safe if they are gender neutral.
Examples: I once went to a toilet in a wetherspoons in Sunderland (oh the glamour!) it was a cavernous space that was accessed by walking down a long lonely corridor in the basement. (300 meters from the bar area?) once inside the toilet, there were stalls back to back, so half of the space was concealed behind the stalls, looking at the room from the door, then there was a large area with mirrors. Whole space bigger than the ground floor of our house. Weatherspoons in Sunderland, not exactly a place I wouldn't expect to meet obnoxious drunks, big gangs of blokes on the piss and on the pull etc. no way I'd feel safe there, even going with a friend.
Whoever designed this toilet had not considered at all how a woman alone might feel going there and had not considered that it would be easy for a predatory male to conceal himself behind the cubicles and equally easy to hang about in the corridor and follow someone in without being seen or heard.
It's no secret that the world is designed for male bodies in size and layout, but also women's specific needs are invariably ignored.
Compare this with the toilet in a community, radical space I use, where there are lots of trans people. Two toilet areas, now gender neutral, but once designed for single sex, open out onto the bar itself. You can see people washing their hands and there is a direct line of sight from the bar to the doors of the cublicles. It is a calm and friendly bar, full of gentle souls. I would never feel threatened there but that's not to say there might not be threatening people, because they don't wear a sign. However, if there was a problem feel confident that it would be noticed and dealt with.