You (maybe inadvertently) have hit the nail on the head.
There are no standard of ‘femininity’ a woman must achieve to be allowed to use female only spaces - she just has to be female.
Likewise, even if a man wears a dress, it is not a passport to female only spaces - he is male.
It sounds like you may have a form of sex blindness, a condition I have very recently observed in people’s claims that they honestly can’t tell who is male and who is female and have to look at how they are dressed and their haircut to work it out.
Thankfully, I think the majority of the population can just use their eyes (and ears) and reliably discern a man from a woman. I can even do it at a distance - go me!!
All men, regardless of their outfit should stay out of women’s spaces. It really is that simple. The teeny minority of men who are in sole charge of their baby can find the unisex facilities with baby changing and plan their journey accordingly.