(excluding me from female toilets in public buildings a la backwoods USA, for example)
What on earth can you mean robin?
Discrimination imposed to deal with illusory fears - no.
You sound very confused.
Your fear of being in a backwoods male toilet is valid, because of male violence?
But if course ours of having men in our private spaces as we disrobe, for the same reason, isn't.
Plus you are using the argument that a woman's group have no problem, because you've never told them what the problems are!
Your personal preference of finding the women's more appealing to you (shock), is neither here nor there when women get to determine their own boundaries.
You violating their boundaries is violating their consent.
You asked up thread what risk you posed, personally. Violating women's consent shows exactly what risk you pose.
And the statistic for surgery comes from GIRES, I believe. A trans support group.