Oddly, I never mind a male attendant in the Ladies'. I know why he's there, he's not hanging around with evil intent and he's keeping it nice and fresh for us. I don't mind a transwoman who's presenting as female slipping in and using the space the way women do: for a wee etc, a hand-wash, a make-up and hair check, an outfit repair, a giggle with a friend, a confidential chat or a private weep. Most of all I don't mind someone slipping in to avoid a male sex pest or angry violent stranger. No issue at all with any of that.
My issue is the conversion to unisex status which prevents all of us - women and transwoman - having our peace and privacy for all the above activities more or less assured.
Yes, of course a violent or predatory man could barge in to the Ladies', but he'd be challenged and help could be sought. If the toilet is officially unisex, he has a right to be there even if he's pursuing a reluctant acquaintance or unsuspecting stranger or a child. He'll get further and do more damage. If transwomen are nervous of using the Gents', how does it help to invite all of the gents in?!
There's a reason lone fathers are apt to lurk outside the Ladies' and ask a woman to escort their daughter in with her. And there's a reason mother's bring their little boys into the Ladies' with them even when they get a bit too old for it. Predatory men exist and you don't know when they're about as they don't wear labels.
Unisex toilets just make everyone equally unsafe and remove a refuge many of us value highly and have needed badly at some point.