Yes I think this is a lot to do with stigma, and the association of femininity with weakness, stupidity, or homosexuality. Men don’t want to wear skirts and dresses not because of the nature of the clothes, but because of the stigma, being laughed at, being feminine when it’s important for them to demonstrate that they are masculine. (And I don’t think that in itself is wrong - we can choose to use gender norms to emphasise our sex, or not. How we use gender serves various semiotic purposes)
For men who like cross-dressing / have AGP tendencies, from what I understand this shame and stigma is part of the thrill, the idea of being weak and dominated. All of this is the effects of sexist gender stereotypes.
But then we do have men who do do feminine-coded dressing and make-up and that is socially acceptable in particular circles in various ways - gay culture, music and arts, and youth trends like the glam and androgynous trends of the 80s.
And in a society that says “men can’t wear skirts” we actually do have kilts, clerical and monastic robes that are effectively dresses, and men from several Asian cultures who do wear dress-like garments. And these are fine and not shameful because they are coded masculine. Kilts are actually very masculine socially and if you go to a traditional Scottish wedding with all the men in kilts it just feels totally male gendered and not odd at all.
I bet loads of UK men would be happy to wear a flowy, cotton djellaba type thing when it’s hot, if it was masculine-coded as it is in some places.
The other way round, women wearing trousers was scandalous (to some) at first but essentially they were practical for various activities and an escape from stricter dress codes. Now we have loads of options as women including androgynous or actual menswear, and feminine-coded versions of trousers and shirts etc - but none of them are really about presenting as a hypersexualised, scantily clad stereotype of a man, it’s about wearing what works and what we like rather than fetishising masculinity. That’s what needs to happen for men too - dresses and skirts that they can wear because they work for them, without stigma. I think if and when that happens, it will reduce the appeal of AGP too.
Perfectly masculine, everyday-outfit bloke in dress:
https://www.gettyimages.co.uk/detail/news-photo/man-in-djellaba-playing-with-his-mobile-phone-on-the-news-photo/481741249?adppopup=true