I think a lot of this stuff about passing can be simultaneously true: that a number of trans women are widely read as female in their daily life, such that they do experience daily misogyny (for much of this, you only need to be read as female by men, after all), and also that a number of TW don't pass as well as they think they do because the women who encounter them give no indication of clocking, since to do so sits somewhere between unkind, unnecessary, and potentially criminal. I get that 'privilege' is a contentious word here as it's been repeatedly employed to suggest that natal women are always privileged in relation to trans women and I do think this is bollocks, but at the same time I can well believe it is easier if you are able to fly under the radar rather than be read as trans.
However... at a policy level, I don't think we can or should discriminate between trans women who pass and trans women who don't pass. Even if we accept that there are trans women, perhaps including Butterfly, who pass always, even when among hypervigilant women and would therefore cause no consternation in a women's rape trauma recovery group, we don't (as I understand it) have a lawful mechanism for allowing these women in but keeping out the Alex Drummonds of the world: trans women who do not pass and do not claim to pass, but who we still can't bar on the basis of being actually abusive as per Karen White or the Canadian with the grotesque waxing demands. This is a real world problem and handwringing on the Internet won't fix it; natal women must be able to use the single sex exemptions as provided by the EA (as I believe/hope Starmer has affirmed...).
I don't agree that the existence of trans men, non-binary people, people with DSDs and butch women significantly muddies the waters. I would support the inclusion of female-bodied NB people, and women with DSD conditions, in women's single sex spaces; I think butch women are read as men less frequently than is claimed in these arguments (an accidental "sir" followed by a swift apology, not unusual, but a protracted belief that a butch woman is male, much less common) and fundamentally that is about incorrectly identifying a person as male rather than correctly identifying a person as male so the implications are very different.