Of course, you don't have to have been raped or sexually abused, or to need, wand and appreciate women only spaces when it comes to 'intimate' public facilities.
You can see and experience being a woman as a great strength and still appreciate that the realities of sex and the particular vulnerabilities of female biology mean that women are best served by single sex spaces, services and sports.
However, It is not just about safety and protection, but about the comfort and dignity of one's sex, too.