I'm so deeply sorry to hear of your experiences. And I'm especially sorry to hear that no one intervened. There have been awful cases of very public attacks that have gone on uninterrupted - I think we're all aware of that. It's unthinkable.
These many distressing instances of a lack of intervention still do not, to me, though, represent an argument for removing what protections we do have.
On a population level, the stats posted above show that transwomen are significantly more likely to commit sexual offences than so-called cis-men, and that women are significantly more likely to be attacked in spaces that males can enter. I honestly can't see a strong argument against retaining single-sex access for women in the light of this.
You use some quite absolutist phrasing above - "naive and wrong", and "not true". There are few such absolutes in this debate (in life?!) and the use of them always makes me somewhat wary.
What I would say, for my part, is that...
Overall, though, the stats indicate that, on the balance of probabilities, women are safer when all males are unambiguously denied access. As such, my concern is with anything that substantially undermines this standard, which is why I mention transwomen. I highlight the stats re. transwomen's crime only in response to repeated assertions that they don't present at least the same risk as other males, as justification for their being permitted to dissolve the previous social contract of single sex spaces - the evidence, in fact, suggests that this is not the case. This may be for any number of reasons, including cynical opportunism by cis-men, but the outcome is the same.
As regards the emphatic dismissal that i's "simply not true" about people being apprehensive about questioning males in female spaces the current context, again, I'd love to have your certainty. Here, I don't have reliable statistics and can only rely on personal experience, and very wide reading of books and articles on this subject from a range of perspectives. I think that I'd argue that, when the Victorian government in Oz publishes a video campaign overtly shaming women for hesitating to enter a lift with a lone male, and the ACLU and Amnesty openly censure women for similar, I have fairly solid foundations for believing this - certainly reasons that are far removed from a "media frenzy". My sources do include the Daily Mail, too, though, as this represents another key reason for my thinking this: witnessing leftwing sources like the Guardian - and even the BBC, in one recent, memorable instance - target journalists (eg. Hadley Freeman) who dissent. Not to mention following multiple court cases in which women have been condemned as phobic for requesting rights unquestioned a mere few decades ago. This does, to me, add up to a culture of fear. To return to personal experience, I also know that, to my shame, I once babbled a desperate "transwomen are women" as, quite frankly, a kind of instinctive Hail Mary to ward off an angry teen! It shocked me to reflect on it afterwards. I often think of it.
NB. By referencing the university sign, I don't necessarily mean to imply you agree with this, or, by referencing the toilet attack, that you posted this yourself.
I hope you're in a better place now and appreciate your lengthy response. I hope you don't mind my similarly lengthy one (having been chastised for this earlier!) Unfortunately, these things need such thought and care, because, as you say, they're really, really not "simple".