If it's ok to require women to self exclude from facilities they can't access, then it's ok to require men with TQ+ identities to self exclude from facilities too.
Otherwise your middle ground is actually just prioritising men over women, which is not really 'middle ground' is it?
The middle ground would be third spaces, gender neutral. Those happy to use mixed sex facilities or who do not feel ok in sex based spaces have their needs met, and those who can only access single sex spaces have their needs met, no exclusions. This would however require male people to respect that women need inclusion too, and not to insist on controlling and using female only spaces.
You know, like as happened in Hamstead Ponds, where a male who claims a TQ+ identity can take their best choice in the moment from all three ponds, male, mixed and female, where many women who used to love swimming now cannot go at all. Because of selfish, exclusionary males, for whom two accessible choices was not enough.