I think a lot of it is rooted in the unwillingness to acknowledge the basic truth that none of us have access to what it 'feels' like to be someone else. 'I am really female/male inside the wrong body' (which was always understood as a metaphor, but now seems to have become literalised) - what that really means is 'I have created a fantasy of what being female/male would be like, and I want to become/live/act out that fantasy'. But that is quite literally true, right? If you are one sex, unless you have invented a magical telepathic/empathic device that enables you to get inside others' heads, then any conception of what is is like to possess someone else's lived experience is a form of fantasy. I don't get to know what it would be to be black, no matter how many times I read Toni Morrison or watch the Cosby Show: I might claim I 'feel like I am really black', but that feeling is only ever based on an act of imaginative identification. No matter how much I want that not to be the case. There isn't really any way around that: I don't get to claim that no-one else in history can visit inside others' heads but somehow I can.
The denial that wanting to be the other sex and 'feeling like' the other sex is a form of imaginative identification (however much is is wanted or dearly felt) ends up in the loopy ideas about male and female brains - which clearly don't make sense, since brains aren't sexed in and of themselves, of course they are already sexed, but only by virtue of being part of the sexed body that they are already in! It's a bizarre form of dualism to have this odd concept that the brain floats freely within the body somehow without being part of it.
Why not just say 'I want to be a woman (or man), because my image of that would be like fits much more with my self-identity. I would like to be accepted in society as a woman'? Great! Everyone can get behind this. I would like everyone to feel happy with themselves, trans, male, female, gender queer, agender - everyone.