That's right.
Many of these men may not want to actually assault the women in their own spaces, but they are absolutely using them for their own ends. The women's presence is crucial.
And, as demonstrated, not just the women's presence, but their exclusive presence.
There is no validation that this man is a woman, unless they are in a place that is exclusively for women. And the more unequivocal that space is, the more they want to be in it. Hence breastfeeding groups, stillbirth support groups, and rape refuges.
Men are using women on a conveyor belt of convenience to validate and/or arouse themself. (Not all of them, of course, HQ).
India Willoughby notoriously boasted about walking past numerous unisex toilets at an airport, to find, in the back of beyond, a woman only toilet to use.
Because the merest possibility that it's not exclusively for the use of women only, dilutes the validation.
It's got fuck all to do with a place to pee.