That's what I see when you make those arguments about trans women. Every argument you make in favor of sex-segregated spaces could just as easily be an argument for the exclusion of gay people.
You could, but the argument would be wrong. Whether an argument is true or false depends on the premise.
The argument here is "if an identifiable group is much more prone to violence, then, on balance, it is beneficial to segregate them from the rest of the population, in some circumstances, despite it being a form of discrimination".
To apply the argument you need to identify groups satisfying the premise.
Males are such a group - over ten times more prone to sexual violence than female people, and towards females most often. Gay people are not such a group. Nor is there any other group I'm aware of. This difference only arises in male-vs-female.
So it makes sense to exclude all males from female spaces.
Even if you could demonstrate a subset of males were lower risk - and the stats make clear that trans males are not - adding exceptions makes the general exclusion unworkable, unless the exception is clearly identifiable
There is one semi-common exception, to aid mothers, and that's boys under a specific pre-puberty age - not too unworkable because puberty is easily observable.