Its in the face, nothing to do with build or clothing.
I saw a young bloke, nice looking chap, in 'womans'' clothes on the bus, full make up and long hair. (I suspect from his manor he though he 'passed' (he looked lovely btw))
Mens faces and womans faces, even though it must be only millimeters difference in the jaw, forehead, the set of the eyes, but its immediately obvious.
I wonder if, due to evolution and women being at such an elevated threat from men, that men can't tell so easily.
But actually that does not hold water as men are also at a hugely elevated threat from other men compared to women.
I think everyone can tell.