Yes. I've heard men use it scornfully on a very few occasions, usually about clerics, as an example of their contempt - for a male to be in a dress for them is a noteworthy insult precisely because of their feelings about women.
If I have ever used it, and I don't suggest I've never said it, it's not been a pejorative, it's been a simple descriptive when a person is attempting to convince me of something which isn't true and requiring my agreement.
So, in those cases, as when you are trying to convince a child that Father Christmas is sitting in the grotto ready to hear their Christmas list and they say, "That's not Father Christmas! That just Uncle Des with a pillow in his jacket!" I may well have said "That's a man. In a dress."
The operative here is not actually on what the man may be wearing. It's a phrase expressing that I understand that whatever artifice may be overlaid, I can see the facts a mile off.
I've not said it to anyone's face, usually it would have been if I've been shown footage of someone. It's rarely needed, tbh. Most people would just say something like "Pull the other one." Or "Don't be such a silly sod" to the person who informed them of the 'alternate reality' they were supposed to buy into.