In my view, if a man wants to sexually assault a woman, he will find a way to do it.
I mean, by this logic - if a man wants to sexually abuse a child, he will find a way to do it, so let's scrap CRB checks on people applying to work with children because most people aren't paedophiles and they're a waste of most people's time.
Abusers have jumped through all sorts of hoops to gain access to potential victims - that's not an argument for removing the hoops.
If you want to make some kind of cost-benefit risk analysis argument, then go for it, but you need to have real, robust, data on the risks to everyone involved. But what if your data says that women have to endure an increased risk in order to reduce the risk to others? Don't you think women deserve a say in that?
Actually, as a post-script - I'm comparing male paedophiles to male rapists above, not trans people, in case anyone feels like telling lies about my post.