Your point that the 'The trans community represents a tiny, tiny proportion of the population' is very true.
Since that is the case, do you ever wonder why on earth such a tiny, tiny proportion of the population demands that the vast majority of the population accept changes in the law, medicine, education, the English language, the media, etc. etc.?
And on the specific subject of toilets, do you ever wonder why a tiny, tiny proportion of the population, who are perfectly capable of using the toilets designated for their sex, are very aggressively demanding that special toilets should be added to the existing women's/men's/disabled people's provision, regardless of the cost and disruption?
Why shouldn't that tiny, tiny proportion of the population just go about their business, not hurting anyone, by respecting single-sex spaces and using the toilets designated for their sex?
If there is scare-mongering going on, it's not working on me - I am not particularly scared of a transwoman using a women's toilet, no more than I would be of any other kind of man using the women's toilet.
I just believe in the principle of respecting other people's spaces; any man, trans or not, who uses the women's toilet should not be there, on principle.
They don't have to frighten anyone or attack anyone to be in the wrong - by failing to respect a single-sex-designated women's toilet, they have removed themselves from the category of 'people just trying to go about their business, not hurting anyone' .
They have revealed themselves instead as people who stubbornly insist on going where they know they don't belong [cf the UKSC ruling, for the avoidance of doubt] and where they know they are not wanted.
That's just wrong on principle, scary or not.