"Disability is the perfect example because disabilities are socially defined."
Not randomly, though. And not just on the basis of, "I feel like I am disabled." There are studies done to show that a condition is indeed a disability, including in the case of invisible disabilities.
Not only is there no such study about trans people, but there cannot even be. What would such a study observe? Are trans women biological female? No. Then what else? Being a woman is not about behaviour, or interests, or anything observable or measurable.
So really, your analogy is like if a random group of disabled and able people got together and decided that, "Having blue eyes is a disability." Predictably, other disabled people would go to court to fight this, and in the end, it would have to be clearly stated that no, having blue eyes is not a disability.
"For a long time, those with invisible disabilities were not considered allowed to use disabled facilities. "
Which wasn't fair because they do have a disability. That's not at all the same as not having a disability and self-determining that you can go into the disabled toilets anyway.