As a society, we are already well aware of the lengths sexual predators will go to to access their victims. Why would anybody think that allowing men into women’s and children’s spaces wouldn’t attract these dangerous men?
Not only that, but it is becoming clearer that some extremely successful men with sexual fetishes (why would it be restricted to society’s losers?) have been driving the narrative via WPATH, Stonewall etc. As NotBadConsidering says, there is a group of men for whom the idea of a cohort of young adults, legally able to consent, but with essentially children’s bodies, is a very exciting prospect.
Anybody, with any degree of power, who is enabling this (be it puberty blockers or allowing men into previously forbidden spaces) should think long and hard about who this benefits, and what, exactly, they might be getting out of it.