Sex is defined by large or small gamete production.
Brains have nothing to do with defining what sex is because many species are plants and don't have a brain.
One of the things that annoys me about the 'sex is not a binary' argument is that it relies on telling people they are not clever or educated enough to understand the science.
Which is actually what people have always said to women.
The science is straightforward, and sex is a binary.
Sex is a binary in all multicellular sexually reproducing species. Sex is binary in plants with both male and female systems. There is no third sex system.
It wouldn't matter if every single human had an intersex condition. We would still reproduce as male or female, just as all intersex mammals and birds do.