Sex is about the role you would play in procreation if you procreated.
Well over 80% of the population are fertile and of the remaining 20%, I doubt any are unclear as to whether they would be seeking to inseminate or gestate should any infertility issues be resolved and they want to procreate.
Gender fluid people do not use different types of contraception (those for males or females) on different days, in line with their identity.
A sorting hat is not needed to work out which cross sex hormones should be prescribed to those wanting to 'physically transition'. Observing their sex suffices.
Sex really is binary and not difficult to determine. There can be / are differences and anomalies between people within each of the two categories, but this doesn't sit those people outside of being one of the two categories.
Where there is a legitimate reason to segregate people on the basis of sex / sexed bodies, it makes no sense to then determine 'entry' by identity or any other criteria than sexed body.
This all boils down to some groups wanting to uphold having sex segregated spaces but for access to be determined by identity. They do not want identity based spaces, or no segregation at all.
They also know they will get nowhere by admitting that they want spaces to be, on principle, segregated by one set of criteria but for access to be determined by a different set. Therefore, they resort to trying to 'queer' sex categories.
If they really thought sex was so individualised and non binary, they would focus on challenging the legitimacy of sex based spaces. This is where there are conflicting interests within the trans community between mtf and ftm and those who are 'non binary'