I've been questioned in women's loos for years! Well before these debates were mainstream about trans people.
I also get ID'd all the time because incidentally I also look about 12, a 12 year old boy on a bad day! I am a 34 year old woman with a child!
Getting ID'd is fine, I actually had to show ID for Calpol not long ago. Which was quite funny.
Being questioned about gender in a public space is fine when kids do it it, I just say oh I have short hair and they're fine. Also kids are kids and question they don't tend to accuse.
Adults will often start off more aggressive (verbally not physically) or lots of uncomfortable stares, sometimes I've had it where people don't believe me even after I've spoken! I don't have a deep voice. Fairly tall but I'd say extremely regular hands although never seen anyone look at my hands they stare at my face and my hair!
It's deeply uncomfortable being stared at or questioned in changing rooms/toilets because I don't want people to think I'm some pervy guy in the wrong place and my own issues play a part in questioning by strangers and confrontation being a problem.
It doesn't sound that bad on the face of it but knowing that in any public space I could be challenged and confronting is tiring and unpleasant.
None of this is to say that women should 'roll over and accept men in women's spaces' it's more just to raise the point that identifying people by the way they look isn't as easy as you say.
As mentioned, the solution is a third space or just individual cubicles. A third space everyone can access but also respects the single sex spaces and the assumption is only bio women are in the women's space. The relief I feel when there's individual cubicles/gender neutral means I won't have to encounter an awkward conversation or challenge for the whoever knows how many times when just trying to have a wee!
Others need single sex spaces - so these should also be protected. I'm not in disagreement with that, but just have a different perspective of someone who is often challenged.