When in doubt retreat into straw-manery.
"All those trans-people are nasty men planning to rape women." said precisely no-one, ever.
What we have said is transwomen are, biologically speaking, men. That's just a fact. Damn that biological reality for being so transphobic.
Also, women are, understandably, often very wary round strange men. We have good reason to be. One in four of us will be raped or sexually assaulted at some point in our lives, 98% of sexual offences are carried out by men, round about 6% of men will admit to behaviours which meet the legal definition of rape.
If I am offered single sex accommodation by a train company overnight in a confined space with just one other person, I expect that accommodation to be exactly as advertised - single sex. Not single "what gender I feel today."
And funny you should bring up the other 99% of the world. You might want to read up on the campaign for women's toilets (not women plus third-sex men, women, as in the people with vaginas) in India. There is a massive problem with women in India not being able to participate in life outside the house safely because they have nowhere safe to go to the toilet.
www.bbc.co.uk/news/magazine-28039513