Sex in mammals is both fixed and also binary. It is defined by the type of gamete (there are two types, small or large, hence binary) that an individual’s body is organised to produce. The male body is organised around the production of small gametes, the female body is organised around the production of large gametes. In the vast majority of cases, the sex of an individual human, as defined above, will correspond with their expected sex chromosomes. Also in the vast majority of cases, external genitalia will unambiguously correspond with sex, as defined above, from a relatively early stage in pregnancy. Rarely, an individual may have atypical sex chromosomes (usually three instead of two) producing disorders in sexual development, which may render them ineffective at producing their expected gametes. This is usually what people mean when they use the inaccurate and outdated term ‘intersex’. Similarly, there are rare cases in which external genitalia does not develop in the expected way, leading to ambiguous sex at birth. However, neither in the case of unusual chromosomal makeup nor in the case of ambiguous external genitalia is a third type of gamete produced. Affected individuals are still either male or female, it is just more complex to categorise them than in the majority of cases. (Gender activists like to talk about clown fish changing sex and sex mosaicism in birds, but I think the platypus is a more useful reference point: despite laying eggs and having both something that looks like a beak and also venomous spurs, features we usually associate with birds or reptiles, the platypus is nonetheless a mammal because it feeds its offspring with milk from its mammary glands. The classification isn’t as obvious at first glance as in most cases, but it is solid all the same.)
At any rate, it is unclear what relevance these rare cases have to the trans debate. The men asking to be called women are not affected by chromosomal disorders, they are uncomplicated males, some of whom have not merely successfully produced small gametes but have even successfully united their small gametes with large to make new people. Similarly, even if we said that the platypus was simply too complicated to place within just one of the usual groups, that wouldn’t collapse the whole system of grouping. We wouldn’t think, just because a platypus lays eggs, that a pig might be cold blooded, or worry that a crocodile could somewhere out there be suckling her hatchlings.
So it is really pretty simple, at root. No woman can have testicles, because this organ produces small gametes and is therefore male by definition. No man can have ovaries because these produce large gametes and are therefore female by definition. If you chop off a male’s testicles, he does not stop being male, because his body remains organised around the production of small, not large, gametes. What you now have is a damaged male. If you cut out a woman’s ovaries she doesn’t stop being female, because her body is still organised around the production of large, not small, gametes. What you now have is a damaged female. In the same way, if you (cruelly) removed a cow’s rumen she wouldn’t stop being a ruminant, she just wouldn’t live very long. If you cut off a platypus’s mammaries you wouldn’t turn her into a bird, you’d just make it impossible for her to feed her young.
This is all incredibly obvious stuff. It requires only cursory thought. It may disappoint or upset some people, but a fact doesn’t cease to be a fact just because it makes people sad.