I understand the point that you're making, but that's why the public debate about this needs to be honest about the fact that the impact of people using spaces for the opposite sex isn't equal both ways.
Men represent a threat to women and women do not really represent a threat to men.
So if a passing trans man is in a women's space, the women in that space will perceive the threat to be the same as if an actual man were in there. And the perceived threat they experience is harmful in itself because of the distress it causes, even if there is no actual threat.
If you flip it round, a passing trans woman in a men's space might make the men in that space feel uncomfortable, but they are not going to feel that their physical safety may be at risk. They represent much more of a risk to the trans woman than the trans woman represents to them.
Feeling discomfort is not the same as feeling unsafe. (And I mean actually unsafe in the usual sense of the word, not claiming to feel "unsafe" because someone does not share your views.)