Sure, no one is stopping them, at least outside of uniform related situations.
However, I thin the reality is that a majority of people like to have some level of differentiation of fashion for men and women. Maybe they don't really care what that entails, at least in a principled sort of way, but they like there to be a difference.
As long as that is the case, there will be some clothes or other accessories that are associated with a particular sex, and and if a person of the opposite sex wears them, that will be noted as unusual and possibly as an attempt to send some message.
I don't see it as a big deal though. No one really needs to wear anything in particular. I seriously doubt that 200,000 years ago people felt less authentic because their choices were a deer hide or a bit of mammoth fur.