Yes there's a difference between men who wear feminine clothing for style reasons, because they're experimental/creative people who like mixing it up, or because it's in fashion, like in the 80s - and men for whom wearing "women's clothing" is a fetish. These men are now being included in the definition of "trans" by the likes of Stonewall, and many of them get an even bigger thrill from being considered actual "women" and being told they can now invade women's spaces with impunity and get a further thrill from the validation and/or transgression and exposing themselves - hence all the videos of men in porn-style outfits masturbating in women's toilets etc.
But getting a sexual thrill from this is an extremely male behaviour. Women don't generally do it the other way round - partly because they tend to be less sex-driven and entitled, and partly because women wearing "men's" clothes ie trousers or boots has become totally normal and mainstream so isn't transgressive.
I think there's a strong argument for normalising men wearing feminine clothes in the same way, i.e. because they like them for style or comfort reasons, as this would reduce the fetish element that is linked to the idea of humiliation and transgression.
After all plenty of men around the world wear dress-like and skirt-like items, especially in hot places, and it's normal not a fetishy thing.