Either you accept most pass and will be indestingasble from the sex they present as, or you don't and male toilets are a danger to most trans women.
Most transmen pass relatively rapidly, once on testosterone, and tend to slide from being perceived as a butch lesbian to a bloke. Transwomen tend to go through an elongated stage of looking 'obviously trans', and some never fully pass, particularly those who transition later in life.
Testosterone is quite good at doing things like adding facial hair and deepening the voice. Estrogen, however, does not stop facial hair growing in the same way, or affect the voice box. On the other hand, bottom surgery for transmen is less well developed than the equivalent for transwomen.
You could never, however, put into law a system where only people who pass get to use the facilities of their acquired gender (who gets to decide if they pass or not?) or one where transmen and transwomen are treated differently - there would be a very swift legal challenge.
That’s why people argue for a third space.
And who's going to pay for that? Will you be personally chipping thousands in to have an extension built at your local swimming pool to house a set of changing rooms for trans people? What about small businesses with toilets?