My dad used to wear skirts quite often in summer. Cotton wrap ones, like you buy near the seaside. No one ever batted an eye, though this was possibly because he was very, very masculine seeming and an agressive alcoholic. People who had batted an eye would have had a newly broken eye socket around the eye... It genuinely didn't look wierd though, it fitted in with his generally grungy, new age tramp look.
Conversly, my mother dressed almost exclusively in mens clothes for many years, which makes sense as she has very broad shoulders, is tall and needed clothes suited to physical work.
People should wear whatever they want, so long as it meets all requirements of decency for the situation.