It's not logical, but I do think there is an element in your question, OP, that's maybe a bit of an assumption.
Which is that all the things that signal gender, like clothes, are really what people are interested in. So a man who prefers dresses is really just interested in dresses, and if somehow if dresses could be equally worn by anyone, they would be better off. (And I know I am being a little reductive about it just being about clothes here, I'm just trying to keep my example simple.)
I don't think that's really what is going on at all. While some people may not care about signalling their sex through clothing, and some people are even a little uncomfortable about it, the reason for things like gendered clothing and certain other cultural attachments to sex is largely that a lot of people do like to relate their presentation to some extent, some of the time, to their sex.
Using myself as an example, a lot of the time my clothing is pretty neutral, even if I wear a skirt it's a practical choce rather than anything else. But some of the time I dress in a way that is more overly feminine, and the reason I like that is because it's overly feminine. Occasionally I even like being more masculine and the reason I like that is that it's more masculine. Femininity or masculinity without a reference to sex are meaningless, even if we "gender bend" and move outside of that.
So without getting too far into motivations, the whole point for the people you are talking about is about femininity or masculinity, and if dressing like a woman means wearing a dress, or wearing mukluks, that is going to be what they look to do. Wearing a gender neutral dress would not be of interest.
IMO the idea that most people might one day decide to give up on noticing or wanting to have clothing or some other cultural signals about sex is pie in the sky. Most people like these so long as they aren't terribly restrictive, and I don't see that changing, unless we somehow get rid of sexed bodies altogether.