Lots of people generally think that people should use the toilets/facilities of their biological sex.
This would mean transmen using the women's bathroom. Lots of transmen 'pass' as men, which means if they're in the women's toilets then people will fell unsafe and uncomfortable with a 'man' in the women's toilets. You wouldn't know if they had a penis or not. They're just a man in the women's bathroom.
Lots of time you really can't tell a transman is trans not just a man if they've taken hormones.
Birth certs and GRCs and genitals aren't checked when you enter a public space.
This could prompt men claiming theyre transmen to enter women's spaces too, no?