So if a transwoman, who looks more feminine than me goes into a ladies loo, or a fitting room, would they get challenged? But me, as a female but looking more androgynous / male (at times), when I do get challenged, is it about appearance? Because unless someone's going to be asking me to pull my pants down so they can check my lady parts, no one is ever going to know.
It is a lot about looks. People are lazy, some seem to go exclusively by haircut and clothes.
And I, too, wonder whether people challenge you not because they're actually not able to tell what sex you are, but because they want to give you crap about not conforming to gender stereotypes. I have read about butch lesbians with very obvious breasts and female voices being told to not use the women's loo; that can't be anything than intentional bullying of gender non-conforming women.
I am not a very perceptive person, but I would look very closely before telling someone they are wrong where they are. And in 90% of cases, if the person hasn't taken hormones, one can tell someone's actual sex when one really looks. (And to be honest, if an androgynous person with short hair uses the women's loo in a normal way, I'll just assume the person is a woman. Or, I used to. With the whole trans debate, I get a bit paranoid)
I suppose a transwoman in a skirt and make-up wouldn't be challenged. Even with obvious adam's apple and all. Because that person would conform to gender stereotypes. (And also because male privilege) People wouldn't want to pick a fight, nowadays.
You look to them either like a gender non-conforming woman whom they can give shit over not conforming, or like an ordinary dude who accidentally walked to the wrong loo and is unlikely to get aggressive over being told off. Can't tell over the internet which is more likely.
Whereas, being accused of "transphobia", that's something people think about twice ...