From the OP:
I don't think that all trans people, especially trans women are necessarily sexually deviant or mentally ill.
Women’s rights defenders are not making this argument either. You clearly haven’t been paying attention to their arguments.
The essential point is that changing laws and policies to allow males entry to single-sex spaces only benefits a small proportion of males. It makes a much larger number of women uncomfortable and there is also a greater risk of voyeurism and physical harm to women. No-one is saying it is necessarily a high risk, but WHY should women tolerate any increase in risk?
To expect women to do so is to say that the feelings of a small special group of males are much more important than the feelings and physical safety of a much larger number of women, and if some women have to experience the collateral damage of sexual assault, so be it.
We are expected to upend the whole way our society is organised and the way it operates, even our language, in order to assuage the feelings of these males. If that’s not the patriarchy in action, I don’t know what is.
My final point is that the laws were originally formulated specifically to help a small number of trans people, those with extreme discomfort with their bodies (gender dysphoria). Then came the push to include a much larger group of males. These are mostly cross-dressers, males who are sexually motivated. There is now no differentiation between the two groups which may explain why statistics show that males in prison who identify as women are considerably more likely to be sex offenders than other males.