The only objection I would have is this. If someone like Buck Angel, who passes completely, were to use women's toilets, the intention would only be to cause mischief. Plus, they (in general, not Buck) have a higher propensity, according to the Swedish study, of committing violent crime.
If they pass there's no reason for them to use transient female spaces like loos, gym change, shop fitting rooms etc.