They may have been.
Eddie Izzard clearly thought, years ago, that it was entirely acceptable for him to use women’s toilets to change in when casually cross dressing.
Boy George has said he’s often used women’s toilets as they were cleaner or nicer (can’t remember which).
I’ve seen people comment that drag queens have used women’s spaces and it was “all a laugh”.
So it isn’t just transsexuals who have been using women’s toilets for quite a while. It seems that many men thought using women’s toilets was fair game.
And now some of those men are angry because the privilege they selfishly took and got away with is now being discussed and women are objecting.
Women objecting shows them up for what they were and are, which is thoughtless and selfish. They don’t like what they see in that mirror so they argue and object and pretend to themselves that it’s all the fault of the women who are confident enough to say no.