Sorry, but this is bullshit.
You're right that if harmful men wanted to harm you they'd find a way, but there are plenty of cases where transwomen have utilised being able access single sex provisions to abuse girls and women. It's a fetish.
The trouble is that if there are no safeguards, rules, or laws in place then there's hardly any consequence for their actions. There are no deterrents because they know by and large they can get away with it.
I have experience of this. I have been repeatedly sexually harassed in a female only space by a transwoman and I am the one that's been demonised for trying to raise the issue, forced out of work, and lost an income.
I am the one who got branded bigoted for not wanting a man to video call, record and ejaculate in the cubicle next to me while I was on my hands and knees with hyperemesis so violent, it made me wet myself from the abdominal contractions, meaning I was often in some state of undress. I'm the one that got told it wasn't possible for there to be a man in the womens bathrooms because the man in the womans bathroom hadn't come out as trans, despite being over 6 foot, broad shoulders, 5 o'clock stubble, deep voice, square jaw, adams apple, massive feet and so until they say they are male, they're female. I'm the one that got told I'd be doxing them and outing them if I took my complaints further. I'm the one that was told the authorities wouldn't help me because there was no evidence of wrong doing and all people want to do is use the toilet, and gaslit into thinking I'm hearing and seeing things that aren't there.
If burglars wanted to burgle your home, they'd do it, but you make it much easier for them if you leave the door unlocked.
Does the sign on the door have magical properties that stop those with a penis entering? No, of course not. But at least now we have legal recourse for those that continue to push those boundaries, even though it's still so bloody hard to get service providers and employers on side because of their woeful lack of equality impact assessments, risk assessments and updated policies that reflect the law.