Dido that's because we bundle multiple related questions in that one verb 'can'.
Most people interpret the question as asking whether a skirt ought to be a neutral, non-sexualised garment. We're in hypothetical future preferred utopia when we're answering this question.
The 'but' comes in because people recognise that we've yet to achieve the desexualisation of the skirt in the public consciousness.
I'm all for men wearing generic skirts in normal, everyday contexts, because this is how the skirt loses its "female" connotation; that's how we get to the world where everyone can choose a skirt or trousers as they please.
Highly-sexualised skirts are a different matter.
By the same token, a man who wears granny pants because they're comfy is not the same as a man who wears arse-floss because it gives him a boner and justifies it to himself by saying it's comfy.