I've seen many women abuse other women who adopt a masculine look and tell thwm to get out. They don't know that she has a vagina. Gender stereotyped. Unisex facilities stop this.
I gather, from what you have subsequently said, that you personally have experienced abuse based on your gender nonconforming appearance. For which you have my utmost sympathy.
Indeed, feminists everywhere have been trying to break down gender stereotypes which would cover a situation such of yours. There is no earthly reason why women cannot present in a masculine way, and vice versa.
The problem with your assertion, is that you are ignoring the inherent power dynamic involved in bullying, intimidation and predation.
Largely, this dynamic is comprised of men in the powerful position and women in the subordinate position.
However individual situations, are just that. They do not counter the overall dynamic, which is somewhat mitigated by sex segregation. Because the power dynamic is based on biological sex.
Gender is immaterial. A man who presents as female, is still of the cohort that subjugates women. He would have been socialised male and retain male pattern criminality. Irrespective of whether you can tell his sex, or not.
The criteria for segregation cannot be based on what somebody looks like. Anywhere.
That is a completely different issue to whether or not you think segregation is applicable in all situations.
But if you have segregation, it cannot be based on gender or looks.