I don't think anyone is " shouting" at you.
Being a man ( I assume) you clearly are unaware of why women value and appreciate genuine women only spaces. The genuine sense of relaxation and comfort, and the ability to feel secure when in a state of undress.It is not women who tend to spend their time staring at people's genitalia or fetishising body parts.
The only reason that 'male bodied people' would use a women only space when there are two other suitable spaces they could use - is for entirely self validating and narcissitic reasons. They don't care how women might feel.
If they genuinely identified 'with' women, rather than 'as' women, they'd be far more sensitive and respectful and use the mixed sex space.