I looked at the (abstract for) the paper, Earlywalker. Sample size of 42 is rather small, and it's an old paper by now.
So I looked for a more recent review, and found this, which says:
'For a number of sexually dimorphic brain structures or processes, signs of feminisation or masculinisation are observable in transsexual individuals, which, during hormonal treatment, partly seem to further adjust to characteristics of the desired sex. Still, it appears the data are quite inhomogeneous, mostly not replicated and in many cases available for male-to-female transsexuals only. As the prevalence of homosexuality is markedly higher among transsexuals than among the general population, disentangling correlates of sexual orientation and gender identity is a major problem.'
Not clear how they are defining 'transsexual' from that, but yes, I'm prepared to accept that small structural brain differences may cause the dysphoria in some people.
But there is overwhelming evidence that people ARE the sex they are, and tiny, barely-there evidence that they are 'really' the opposite sex.