And honestly, I personally don't worry much about the threat issue in terms of my own life. But I do believe that I am entitled to privacy in terms of changing, sharing rooms in a hospital, etc. I probably spend as much time with close male friends as female ones and I'd never worry about them being pervy. The same for my dad, or step-dad, or my uncles. And I've seen plenty of naked men in contexts where you'd expect that.
I still would prefer not to get changed next to them and see their man-bits. And they would prefer not to as well, because they have a sense of respect for peoples privacy including their own. There is nothing wrong with sex based segregation for those kinds of things in a society, it has nothing to do with being unequal.
What I don't care about is the hairstyle, clothing, or name, of the person in my changing room. It's not an invasion of my privacy if a female person who dresses in male clothing, with a buzz cut, named John, is in the next locker. We don't sex segregate change rooms or hospiitals etc to validate people's feelings that would be weird.