I think, speaking specifically about toilets, part of it is probably more rooted in a much older time when gender roles were more traditionally delimited and it wasn't considered "proper" for men and women to share the same toilet space?
Now I'm sure there's also part of it that is offering women a sense of protection from being behind closed doors with random men, who could use that private space and proximity to do anything from making inappropriate remarks to being physically/sexually abusive. I guess to me the question is: if unisex toilets became the norm, to what extent would this kind of thing happen more. There's nothing stopping abusive men coming into women's toilets and doing this stuff now, it's not like they're policed. And similarly, this kind of behaviour can and does happen in multiple places other than toilets. Obviously unisex loos would add to the opportunism factor though.
I guess I'm saying that personally, I wouldn't be bothered in theory sharing public toilet spaces with men (and have done in previous workplaces and public places), and by extension trans women, as long as there were no urinals, only cubicles. The question of actual (instead of perceived) safety/rise of opportunistic attacks is a separate thing to me, I'm not advocating that because I'd feel OK with it we should go ahead and enact it into law/culture.