Women managed to change society to the extent that trousers became 'normal' clothing for women, despite the fact that during this period men held most of the power. They did this because there was a huge demand from women to wear this item of clothing, and determination to change society to reflect this.
Since men still hold a disproportionate amount of power, the fact that skirts and dresses haven't become 'normal' everyday wear for men would seem to imply that there isn't much demand from men to wear these items.
So while clothes perhaps "shouldn't" be gendered, perhaps, when it comes to skirts and dresses, it's just what people want. And of course if a man really wants to wear a skirt, he can just do it.