"Does that mean we should therefore give up all laws against these things, security and measures to protect ourselves? No."
Fine ... but how would you actualise such laws, then? I'm damned if I can think of a way. Toilets and changing rooms - even nominally 'public' toilets and changing rooms - are spaces in which private things happen. Legislation is generally about the public realm and can't easily touch what's in the private realm. In practice, that means, for instance, is you can't post a security guard in a women's toilet or changing room who'll challenge somebody by saying, e.g., 'I don't believe you're a woman'. It doesn't work and can't work. It's an absurdity.
Can you imagine the problems that would arise? Said security guard (who has to be big enough and assertive enough for the job - though, nonetheless, presumably can't look in any way mannish) says to a person with ... I don't know, biggish hands and wide shoulders ... 'I don't think you're actually a woman'. And then what? Said security guard demands to see that person's genitals?
Seriously ... think your way through that. The 'cure' is as much, if not more, of a problem than the issue it was meant to deal with. Imagine just one woman - born female - being asked to show that she's female - because she happens to be broad-shouldered or deep-voiced. Jeez. It doesn't bear thinking about. I've known women like that. It's one thing to get mistaken for a man (as this friend was, often), but it's another thing entirely to have to prove that one is a woman, with the force of law behind it.
Folks, I don't think that, in practice, this argument will go anywhere useful at all. The law, and all its available means and instruments of enforcement, are just too clumsy and inept to deal with it. Women's toilets and changing rooms aren't safe places and havens from men. They might feel like that most of the time, but they just aren't. But - again, in practice - it doesn't matter. Transgender women - or even men pretending to be women - are few and far-between. A tiny proportion of those will want to do anything other than use the toilet or get changed. Of all the predatory men there are, few will think going into a women's toilet or changing room is a fine way to prey on a woman. (Really - put yourself in the mind of such a man. Would you? Why follow a woman into a toilet, where there could be other women who could stop you, or at least act as witnesses?)
It's for these reasons, and more, that it's so vanishingly rare for women to get attacked by men in toilets or changing rooms. Really, at bottom, for me: legislate against it, if you want. But I don't think it's going to make the slightest difference in practice. A transwoman will use a women's toilet if that's where she'll feel safest and most comfortable. A predatory man won't care: whatever the law says, he'll only care about getting away with the major crime of preying on a woman. The minor crime of his being illegally in the wrong toilet is not something to which he'll give a second's thought.
At bottom, to me, this debate looks important in theory ... but in practice, it's a nonsense. Most of the time it looks like a discussion about what people should do, while ignoring the ginormous elephant in the room - that of what people just will do, entirely regardless of moral and legal considerations.