I think there are a few good reasons why it is probably not a "a neurological condition".
One reason is that even if all people under the "trans umbrella" suffer from a neurological condition, they quite obviously do not all have the same neurological condition. Heterosexual males who grow up as boys and live as men for decades, doing male things such as competing in men's sports, fathering children or committing sex crimes, before deciding to transition in middle age, do not have the same neurological profile as homosexual males who exhibit signs of gender dysphoria in early childhood or adolescent girls who decide they would rather be boys around the time the realities of female puberty and living as an adult women hit them like a train.
Another reason is that all these different groups seem to be heavily influenced by environmental factors, such as porn, homophobia in their communities, addiction, bullying and sexual abuse, and in many cases are comorbid with recognised conditions such as autism, ADHD and eating disorders. I think it is highly likely that if all these environmental influences and other conditions could be removed, the gender dysphoria would also disappear, in which case it could not be said to be a standalone neurological condition but rather a symptom of other issues.
And finally, with all the time and attention and money that has been poured into trans people and gender affirming care, I would have thought that if being transgender was a neurological condition, there would have been some solid research into it by now which would support this hypothesis. There is much talk of male and female brains, but no large scale studies actually looking at both trans and non trans identifying people's brains. Surely this would be a good place to start, and a fascinating piece of research. You study a few thousand ordinary mens and women's brains to see whether there are any clear differences between the two. If there are not, there's probably no such thing as a sexed brain. If there are, you study trans people's brains in the same way to see whether they most closely match their own or the opposite sex. I do not think the findings of such a study would actually be relevant for the purposes of single sex spaces or sports, but it would be a useful starting point for determining whether being trans is a neurological condition or not. You would probably need to do it by age group because I do not think the findings would be consistent across the board (unless the findings were, across the board, that there is no such thing as a sexed brain). But as far as I am aware no such serious research is being undertaken. Stonewall even campaigned to remove the requirement for a diagnosis of gender dysphoria in order to get a gender recognition certificate, because they want less medical gatekeeping, not more.
If being transgender was a neurological condition, as backed up by serious research, it may be possible to develop a treatment to treat or even cure it. Not affirm it. Because that is what we do with neurological conditions. We treat them. We do not demand that the rest of the world adapts to them. Yes, we sometimes need to make reasonable adjustments for people with disabilities, but there is also an expectation that the person with the disability will try their best to do what is required of them with the necessary support.