Secondary sexual characteristics are biological sex, they are one of the aspects of biological sex. Chromosomal sex is another, genetic sex another. Biological sex isn't seen as one thing, the current craze in gender critical circles seems to be to say "gametes" is the only aspect of biological sex that counts. Biologists disagree as they use the terms I quoted above, I didn't invent them, trans people didn't invent them.
No not a craze. Basic biology.
It starts with the chromosomes. Anything that happens differently thereafter is either a disorder of sexual development or another anomaly.
Brain sex remains sexed no matter what happens.
The gonadal hormones can generate sexual dimorphisms in the brain, but sex differences can, in some cases, be detected before the gonads have differentiated. Further evidence for a direct role of the XX or XY genotype on the brain comes from mice in which the testis-determining Sry gene has been removed from the Y chromosome and replaced on an autosome. In this way, it is possible to generate 'female' mice with an XY− genotype, and 'male' mice with an XXSry genotype. The brains of XX mice differ in several respects from those of XY mice, regardless of the presence or absence of the Sry gene.
And that has a bit about birds hence the chicken above.
www.nature.com/articles/nrn1494