He said that it's wrong to mistrust all men, and if he said that he mistrusted all women I'd accuse him of misogyny.
I think a lot of people struggle with this to be honest. For example, if we knew that poor people were a lot more likely to steal, we would still
be uncomfortable with saying things like, say poor people are more often thieves, and we would think it was wrong to bar poor people from shops, or to follow them around.
In some ways sex segregation for certain things in a little unique. It's true on the one hand that sex segregation in some things is a pretty efficient way to guard against certain types of sex crime. And it's not unfair, men have their own changing areas and hospital wards, and aren't worse off for that.
But there are other elements - one is just that sex is a very powerful and basic drive in almost all human beings. And not all sex crimes are about weird evil people hatching nefarious plans - they can often be opportunistic, something the person might not have really thought about doing, lots of people have pretty poor impulse control. You prevent all kinds of socially unacceptable things and avoid a lot of social problems by just not making that an issue.
But also, it's not the full story to say it's all about risk. Most of us do not want to be changing or living in a room even with opposite sex people we know and know to be safe. It's awkward among adults, in good part because we are aware of the implications of nakedness and sex and attractions. Can you imagine the teen boy expected to change with the teen girl he has a crush on? Would it make him a sex offender if he was tempted to peek, or think about her nakedness? It would be pretty natural actually - why put him in that position?
Anything that applies to opposite sex people still applies when you are talking about self-id. Some would argue if you are talking about fully surgically transitioned gay males a lot of those issues are not so significant, but in any case the trans lobby does not want that gatekeeping so it doesn't come into it.
OP, it seems to me that your husband needs to define more clearly his thinking about sex segregation.