It's not seeing the forest for the trees.
Sex is not defined by chromosomes. People knew about sex before we knew of any such thing. Farmers knew how to breed their livestock. People figured out a lot about plant breeding too.
Sex is about reproductive role. It's about the types of gametes an organism produces, or did produce, or will produce. Or in some cases, where something has gone wrong, it's about what kinds of gametes their body would have made had that not happened. Including errors at the level of DNA and chromosomes.
So if some extra bit gets included, or there is some developmental error in the appearance of the sex organs in the womb caused by a chemical exposure, or some doctor accidentally cuts of a penis during a circumcision, it doesn't change that a particular body would have developed to be a functioning male or female person if things had gone right.
There is no third gamete, nor any necessary third role in sexual reproduction. Just a large gamete and a small one. Female and male. That's it.