And we have been using the women's toilets for years
The sort of gentlewomans agreement you describe has indeed been going on for years and it’s largely worked. It’s now under threat - not from transwomen with gender dysphoria who just want to quietly get on with their lives, but from vocal activists who want Self ID which will mean that any man can enter the ladies.
This puts women at risk, and it puts transwomen at risk too - transwomen are not under physical threat from women, let’s be clear about that - it’s general Male violence that is a physical threat to them. Allowing all men in, and removing the safeguards that kept all men out will destroy the women’s spaces as safe spaces and nobody, not women, not transwomen will benefit from it. Assault rates of both women and transwomen will rise.
There’s also the rather unpleasant possibility of societal backlash - and it won’t be against general men, it’ll be against the genuine transwomen who didn’t cause the problem in the first place. That is not something we want to happen either.
There is a practical solution. There can be Male spaces, female spaces and unisex spaces. If those unisex spaces were large, totally self contained units then they could also be used by families, or mother/son father/daughter pairs, or people who need a bit more space for mobility issues, etc etc.
What can’t happen is what’s happening now, where the men’s remains the men’s and the women’s is turned unisex. That simply isn’t fair.