This issue is a real non-starter.
First of all, most places I know of already have a third, single stall, gender neutral/ disabled accessible/ family room. If places around you don't then you should campaign for those cause they're great.
Also, in most places where people are allowed to use the bathroom of their choice rather than their sex there is a 3rd space made available for those (women) who feel unable to use that space because of the opposite sex now occupying it. The obvious solution would be to make that 3rd space for trans or gender neutral people in the first place.
Second, no one has an issue with a passing transman or transwoman flying under the radar in the bathroom of their choice. The vast majority of the time it isn't an issue. No one is insisting on underpants checks at the door.
The problem comes when we say that the trans person gets to decide where they should be and no one is allowed to object, an also that sex-altering surgery isn't a requirement. That's a problem because you're just saying that men are allowed to use womens bathrooms and changerooms and women who are uncomfortable with that can go f* themselves.
If you're pretty sure you pass, use whichever bathroom you want but respect if you're asked to leave because you're making people uncomfortable.
If you probably don't pass completely (and VERY few do) then use the facility that is correct for your sex. The argument swings both ways: If it is safe for a woman to pee in the stall next to a transwoman then it is safe for a transwoman to pee in the stall next to another man. Any justification for letting transwomen use the womens room is also a justification for why they should use the mens room. I.e. "tranwoman just need to pee and aren't interested in hurting you" same goes for regular men. "Transwomen are largely non violent and non offenders" same goes for men. "If a transwoman did behave inappropriately they would face the same legal consequences for comitting a crime inside the bathroom as anywhere else" well same goes for men.
The ONLY justification for letting transwomen use the womens bathroom as a matter of policy is that you prioritize men over women. That's it.
And yes, we don't have a problem with transmen using the womens bathrooms if they choose. Even if they were to get the occasional side eye they would only have to say they are biologically female to make everyone comfortable (and again, most wouldn't even need that, they tend to get pegged as butch lesbians). Or use the mens room if you're comfortable with that. Or use the gender neutral space. there are so many options, none of which involve forfeiting women's right to safety as a group.