If you choose to widen the focus to include genetic, gonadal, anatomic, and psychological aspects of sex within any one individual, you might be right in saying 'it's not black or white'.
But including 'genetic, gonadal, anatomic, and psychological aspects of sex within any one individual' sounds like simply taking every man and every woman as an individual - some men are anatomically slim and slight, some women are psychologically atypical, some boys behave in stereotypical 'boyish' ways, some girls behave in stereotypical 'boyish' ways, some women are infertile, some men have gynaecomastia..
There's nowt as queer as folk, as the old saying goes
Those individual variations, however, do not contradict biological sex, determined by having XX or XY chromosomes.
This is sufficiently black and white to be the most obvious and practical way to define sex, given that
'Most individuals fall into one of two categories, clearly male or clearly female'
in other words, sex is black and white in the vast majority of cases.
The small number of exceptions are of academic, social and human interest, but they don't alter the validity, and the practicality of using chromosomes to determine biological sex.
BTW thank you TP for responding earlier about the meaning of 'phenotype'.