Why would it? I'm not morally opposed to skirts and I don't think clothes HAVE to be attached to a particular sex - plenty of Scottish men wear kilts, which are functionally the same as skirts, regularly, for example. I also like when my husband wears what makes him feel good.
So if my husband felt good in it I couldn't give a fuck, to be honest.
The only way I could see someone being upset by it is if they think that a person has to dress 'like a man'/'like a woman' which is boring stereotyping. Are you against women wearing trousers, too? Or is that bit okay?