Yeah you have pretty much nailed the confusion.
The single sex spaces question is a bit more nuanced. For example, if Buck Angel walked into a women's toilet, I would feel pretty scared, but if Elliot Page did, I don't think I would care.
Both TG and TS people are mentally ill with body dysphoria, but only TG people think that their illness is reality and don't have the emotional maturity to see that they are the ones causing issues. It is a big feature of narcissism, for example, to act like a toddler. TS people know they are mentally unwell, and don't hold society responsible for fixing that because it's a problem with them, not society.
I think the most dangerous part of all this is Self-ID. Without that, people who want to transition don't have to go through therapy and assessment, which would hugely lower the risk to women. Men who wish to access female spaces because they are peverts would be rooted out early on in the process.
If we could overturn Self-ID, then the risks posed by TG men would go away, and the risks to body dysmorphic teens and non-gender-conforming kids would go away also.