Descartes has to take some of the blame. Transwomen are reliant upon the false idea that the mind and body are distinct entities, otherwise it would be impossible to conceive that someone who has grown up in a man's body could have the mind of a woman.
But that leads us to another misconception - that there is a clearly defined "mind of a woman". This belief is deeply rooted in unscientific gender stereotypes stretching back millenia. For transwomen to be real women, they need both cartesian mind-body duality AND gender stereotyping.
The final misconception is that humans are isolated bubbles who exist largely free of social influence. If there is such a thing as a "mind of a woman" it must not arise as a result of socialisation. If it did, transwomen could not be real women because they're not raised as women from birth.
So, we have two opposing models. One says that women have innately different minds from those of men, but that this is nothing to do with experiencing either the body or socialisation of being a woman. The second says that our minds arise, at least in part, from both our bodies and our socialisation.
It's the second idea that's generally accepted by the scientific community (though there's a lot of debate over whether it is brain / body or socialisation that dominates). However, it's the first that's generally preached by our Anglo-Saxon, neoliberal political philosophy.
Basically, our society teaches a lie: that you can simply choose to be whoever you want to be. It's unscientific, but it's widely accepted. Not surprising as it appease the wishes of many non-trans people too, especially those who wish themselves to be something other than what mainstream society wants them to be. That'll be teenagers rebelling against their upbringing, libertarians railing against Political Correctness and Socialists who are always at war with "the Establishment".
So, why do many people believe transwomen are real women? Because that belief is the inevitable conclusion of a set of values they want to believe are true, even though they're not.
Why do many in the medical community support transition? Because they are not there to judge the beliefs of a person who needs treatment. The current psychiatric philosophy is to make people happy by giving them what they need. It doesn't matter whether or not transwomen actually are women, all that matters is that they have a way of reducing their suffering. Note that the thrust of scientific inquiry is not an attempt to demonstrate that transwomen ARE women, but to find something in their brains that is different from typical men and makes them feel like they should be women. That simple explanation - transwomen have brains that make them feel like women even though they're not - obviously leads to the solution of helping them feel like they actually are women, even though they're not.
But anyway, medics at the chalk face follow accepted best practice, even if that best practice is wrong (read up on cot deaths if you want to see how easily a harmful "best practice" can come into being with no scientific justification).
Well, that's what I think. The reason it feels like we're pushing water uphill is because we're fighting a belief system much bigger than trans ideology. It's the same ideology behind the American Dream, behind modern neoliberal economics, behind the MRA movement, Brexit and Trump. It's Neitzche via Ayn Rand - the will to succeed makes anything possible, is an end in of itself, and will inevitably lead to self-justifying success. And anyone who doesn't have that will to succeed? It's fine to screw them over because they're on the wrong side of history.