Anyway, my initial thought was that only men who are completely outside social norms get challenged, and even then only when an organisation is strong enough to do so.
I think his original premise is flawed, as PP have stated there never really was a carte blanche approach for "passing" transwomen. They either didn't pass and women were scared, or they didn't pass and women were polite. The "stealth" thing is overplayed. Even if a transwoman passes to some people, some of the time, it won't be to every woman all the time.
So I might think X is a natal woman and not notice X in the ladies', but you might notice. Is he really suggesting some transwomen pass so well that nobody notices ever? Especially years ago pre FFS? It seems really unlikely.
It boils down to "women are going to get raped anyway, stop being so mean".
Also, we've never excluded anyone except using local rules and policy. You could walk into the gents' in a shopping centre, it's not illegal. If you did it at work, you might get reprimanded if someone complained and thought your actions were inappropriate. But the law doesn't state that a man can't enter a women's toilet, it isn't even legally required that a sign is put up when a male cleaner is in there, AFAIK. That's just customary.
It's more about reducing risk, and dealing with incidents. If it becomes customary for all toilets to be used by men, then there will be no grounds for (as an example) security staff to ask a man to leave because he is making women feel uncomfortable. There would be no social expectation that he should leave. There is no law that says he has to leave. The risk would increase, because not only will those toilets be used by men who get a stiffy at the thought of being perceived as a woman, but also by men who like to spy on women, listen to women urinate and defecate and change sanpro, and men who want to attack women, and men who hadn't even considered it but are now emboldened by the fact they are allowed. Like how we see the use of prostituted women increasing massively where it is legalised, men will use this new permission because why not? Which in itself increases risk. You might be safer in the gents'.
That's why the "organisation behind us" bit is important. The lack of actual legal provision in many situations is weird. Why isn't it illegal for a man to go into the women's toilets? You could write a law that made it legal for security staff or first responders or police or the cleaner etc to go in there, couldn't you?
And where it is written in law, those laws are ignored, eg in hospitals and prisons. Where the law states there should be separate facilities, those laws are also ignored. The law just isn't working for us.
As for the "poor butch women" bullshit - there is no correlation between crossdressing men colonising women's facilities and women being homophobic to other women. They have yet to explain why they think keeping those men out of the women's toilet makes me safer in any way. (Spoiler: it doesn't.)