It's about the ability to choose, and the question is not whether trans women are a threat, but whether men are a threat.
We know, from policies implemented by institutions like universities, that in practice the only policies that anyone is prepared to implement are single sex, where there is an expectation that people will use the correct facility and can be challenged for not doing so, unisex single user facilities, or mixed sex facilities that anyone can use.
You cannot segregate anything by gender because it is a completely subjective concept, which is why 'trans inclusive' facilities don't attempt to do this.