If you're afraid of men being perverts/assaulting women under the guise of trans identifying and using women's spaces, that's an issue with MEN not trans women.
Transwomen are just another version of men. Fear and distrust is an issue with all men.
Transwomen retain the same criminal rates as men. It almost seems higher judging by some statistics.
Most men are not malevolent but no one can know who is good and who is bad.
Male violence and perversion is unpredictable and no one wants to be holed up alone with wandering strange men of any identity in lavatories and changing rooms etc.
Apart from this though, women want boundaries for reasons of privacy and dignity.
Third spaces should be provided for those who need them.