That might be true in certain circumstances, but I don't think it can be true as a general principle. What about a netball group that requires applicants to have a commitment to playing a regular netball games per season. Assuming men are much less likely to want to play regular netball than women are, they are less likely to qualify for membership.
That is not sex discrimination, because there is no detriment to men: any man who actually wants to play netball can join, regardless of the fact that men as a class are less likely to want to play it.
Similarly, an organisation for 'female gendered people' (again with the caveat: what t f is that anyway?) would be available to any man who had a female gender, regardless of the fact that men as a class are less likely to have this.
You might get into difficulties in some circumstances if the organisation was providing a wide range of benefits in addition to the performance of gender that non-female gendered people could not access elsewhere. If that were the case, then males without a female gender might conceivably be regarded as facing a detriment, on the grounds that they were more likely to lack a female gender than women
But if that were a problem for a female-gender organisation, it would equally be a problem for a sex-based group. Say, for example, Girl Guides was waaay better than Scouts at providing abseiling courses and similar. Could males sue for sex discrimination?? I doubt it