Yes that is a commonly said TRA trope.
'Reminds me of the days of Section 28'.
If these troubled men just used the male facilities, and left women and girls alone there would be no problem. If the absolutely crazed activists let people speak without the bomb threats, the death threats, the turning up and banging windows, banging drums to drown women out, and stopped attacking women, harassing them, getting them sacked or getting their businesses closed down, if they stopped nailing rats to rape crisis centres, stopped attacking the offices of MPs and the EHRC, stopped attacking hotels women book to speak at, stopped the threats of violence towards anyone hosting events...then maybe people might believe it. But it is all just a 'forced teaming' tactic which we see over and over again.
Gay men and lesbians do take exception to the next generation being interfered with by the prescribing of drugs to block their puberty just because the kids like to play with the wrong toys or just to make them 'straight'. That's the link with homophobia.
Not the need for single sex spaces.