I'm 63. Can't recall ever being asked for anything in the ladies by a stranger either.
Re passing: out in town on Sunday, I clocked a person coming towards me which clearly sent out an automatic 'alert' to look twice. It was the male frame and gait combined with a bright yellow dress, three inches above the knee and clumpy platform sandals that (instinctively) looked out of place. Clearly a 'bloke in a dress' as my male companions sniggered. Not very kind of them, for sure, but that was the reality.
I have no problem with blokes in dresses, but if you're in your 50s, like this one, at least try and adopt age-appropriate clothing if you want to 'pass'.
Would I have be comfortable with him in a sex-segregated space, especially if I was the only other one there? Definitely not. He was far bigger and stronger than me and he was just a random bloke, albeit it in a dress, and my instincts would be to treat him as any other random bloke. Would I have challenged him? No - he was bigger and stronger than me. I would have gone elsewhere and probably notched up a little resentment mark on the post because I was the one who felt compelled to change my ways.
That ultimately may be the reaction; tolerance by women will be eroded if their spaces are encroached. Be careful of what you wish for.