Trans women are trans women. They deserve protection, rights and respect but they are not the same as biological women.
That is determined by chromosomes and yes it is binary, resting on the presence if a Y chromosome. There is a small number of people who have sex disorders such as kleinfelter's but these are disorders or anomalies (such as the woman in the oft- wheeled out article who has cells from a twin brother as part of her makeup) and do not denote some type of sex spectrum. Sex is important and cannot be ignored in favour of the much less binary gender. Not in all circumstances.
The point here is that erasure, pretending differences do not exist is not helpful. A more honest approach would be to accept that trans women are not lesser, but they are not the same as women. This approach would allow the safeguarding and fairness issues to be ironed out without this ongoing unhelpful chorus of 'transphobic!' Ringing out any time people want the truth to be acknowledged, which is that there are some situations which necessitate that safety or fairness need to supersede inclusion of anyone who identifies as a woman in all arenas.
OP, the issues you are vague about and have no answers for, prisons etc, ARE the points that are of concern for biological women. You can't be airy fairy 'TWAW, yes there are some issues but these are tricky and I don't know the answers but no, trans women are definitely women and that's that'. If there was no difference then these issues would not exist.
The path to inclusion and harmony is honesty about differences, what these differences signify. and then working out where everyone stands.
Have you read about the man who has detransitioned recently and is suing the NHS? I don't know his full story but it looks as though he was advised by trans rights organisations who I believe are causing a huge amount of harm by trying to erase any acknowledgement of these differences and prevent exploration of the issues surrounding gender dysmorphia, calling this kind of investigation 'conversion therapy' when it really isnt.
Have a think why gender critical women have this stance and what outcomes we really want. I think you'll find few of us want anything but dignity and acceptance for trans women. This cannot be at the expense of women's rights though and it cannot erase the issues that uniquely apply to women.
I also find the argument very contradictory that gender is a spectrum or whatever your wording was, but then in the same breath that trans women should be included in one of two very binary categories. If it's fine to accept that gender is complicated, then why not accept a variety of different terms to accurately describe who people are?