I think the problem is that it isn't symmetrical.
On one hand, women wear pretty much whatever they want.
On the other, femininity is considered demeaning for men so wearing 'feminine' clothes is taboo, and therefore, for some men it is fetishised. I think that if men were more gender-non conforming, this would be neutralised.
However, that doesn't solve the problem that for some men, the clothes are just an avatar, and it's the idea of being a submissive doll woman that is fetishised, and even in a world where there was no taboo around wearing a skirt, mimicking women would still be part of their fetish.