I'm not condemning you or telling you what you can or cannot wear. You are a grown up and are perfectly able to choose your own clothes.
I personally hate skirts and dresses so don't wear them and instead can usually be found in joggers or jeans and a t shirt. So if you are comfier in skirts and dresses, with or without petticoats, then honestly why are you overthinking it so much and assuming women are condemning your fashion choices? Some might, most couldn't give a toss, and either way why would you care what some random people that you don't know might think of your clothing choices?
It's 2026, wear what you want. On a personal you level, if you notice people are looking at you it might be because they think you look ridiculous, but it might just as well be because they think you look stunning. Only you can decide which version of what someone else is thinking inside their own head is the version that you are going to attribute to someone looking at you.
And if someone says it out loud that you look ridiculous then it's up to you whether you engage or ignore.
If you want to talk about women condemning what a man wears shall we look at just some of the things that no doubt the majority of woman on this board has had various men, sometimes from through van and car windows or just passing by in the street, or our male boss etc, tell them:
they look ridiculous / tarty / slutty / a mess / a dog's dinner/ mutton dressed as lamb / a slag / a prostitute / a plain jane / a slob / could have tried harder / tried too hard / skirts too short /skirts too long / that style doesn't suit your shape / did you get dressed in the dark? / you look like you're on the pull / did you wear that for me love? Shall I continue?
If a dress or a skirt is what does it for you, then it matters not what others may think, knock yourself out mate.