80s fashion wasn't sexual in a fetishistic sort of way, but I don't think it really represents a totally non-sexualized fashion.
A lot of the appeal of it was specifically that it was rebellious and counter-cultural, which it wouldn't have been if it were normalized. And there was a kind of sexual frisson associated with that too, because of the boundary pushing.
There are all kinds of male dress that aren't pants that are common in some settings. Kilts, sarongs, robes of various kinds. Some we see in the west and they are very obviously male clothing.
I think it's really notable that for the most part, you don't see the "men in dresses" guys going for these. If anything it can be rather manly men who adopt things like the utilakilt. No one thinks the older African gentleman wandering down the street, or a priest in a cassock, looks feminine. (In fact, when women wear those garments for men they usually look rather dumpy.)
Dresses I would argue are women's clothes cut for women's bodies. Men interested in fashion aren't likely to wear them because for 95% they will be ill-fitting and look awful. Men who do wear them without sexual reasons are often in the music business or otherwise trying to be subversive or attention seeking, which depends on them not being male clothing.
You could, very easily, design dresses for men, but they would not look like women's dresses. And so my question is, if there are really men who just really love the idea of dresses, where are the men's dresses? If they will buy them, people will sell them, and it's not happening.