This is often used as some kind of "gotcha".
Like the occasional woman who has gone trans is more of a threat to us than the mostly intact males who would be entering under the alternative rule ie gender identity.
We have to fix on one of the two and we are going to have to take the hit on the trans men in order to keep the spaces single sex.
In fact, most trans men are still easily recognisable as women. They may look masculine in photos but in real life they are often slight, with a female gait and stance due to the shape of the pelvis, small hands and feet, no Adams apple, smooth skin......