Thanks for putting into words why the toilets and changing rooms arguments make me so uncomfortable.
It feels like a bloke staring you down with a smirk saying "No? So how you gonna stop me luv?"
Prison and sports have male-dominated rule makers and enforcers. The smirker knows he will be forced to obey them. Toilets, gyms, changing rooms have no powerful males on enforcement duty.
The argument that it was OK in the past made me think "Yes it was OK, what has changed for me to make me so uncomfortable now?"
Fifteen years ago I would frequently shared spaces with people who would probably be called trans or non-binary now (like obvious males dressed as a sexy lady, very butch women, people maximising the androgynous look). It never bothered me much. I think because people cared about making each other feel comfortable and who was encroaching on whom.
For example, I was mates with a drag queen, if I had told him (he called himself him mostly) the women were uncomfortable with him using the ladies (we did go clubbing with observant religious women - modest dress, no drink, lots of dancing) then he would have asked what he could do to make it OK and would have gone to the gents or waited until the loos were empty in a heartbeat no bother. Absolutely no entitlement, no smirking, no "Screw you, I'm coming in anyway".
Some vocal recent incomers seem to feel entitled to take, to push women aside. If we say we wouldn't feel comfortable with you in there they don't say "I'm so sorry, I'll use the gents or wait until nobody else is in here, what would work for you?" they say "no point making that rule sweetheart, I'll do whatever I want and how you gonna stop me?"
Then we end up talking about how we can only have a rule if we have a fierce enforcement method (genital checks!!!).
It's all about ignoring women's "no" unless there is a big man to enforce it.