I can't see how to balance the rights of everyone as it stands
Wanting legal entitlement to a group's provision and resources by subjective choice, while not having the characteristics of the group, is not a right. It's a want. It should not be a right.
It will not balance, because men cannot be given this entitlement without massively affecting the interests and sex based rights (actual rights) of women.
It worked after a fashion when this was about a tiny proportion of men with a diagnosed condition and very tight, demanding gatekeeping. It no longer works because it's now about any man, at any time, with no gatekeeping at all.
At this point, no. Male access to female provision under any circumstances needs to end. Instead, third space gender neutral (or in prison situations trans specific) spaces and provisions need to be the alternative available for people choosing not to use the sex segregated provision.