the issue here is here are many trans people who are perfectly lovely.
Are you willing to take this argument all the way or not? Are you going to say "there are a many men who are perfectly lovely", and so there should not be separate male/female prisons, or indeed any other male/female splits (whether you do it by "gender" or sex?)
I'm not sure you've even considered that properly, as I've seen contradictory indications where you've denied both positions.
But surely in your ideal non-discriminatory world we wouldn't be arguing about whether transwomen were classed with men, as we wouldn't be discriminating between men and women anyway, so it wouldn't matter.
I do actually somewhat sympathise in that yes, discrimination on immutable characteristics is unfair. But sometimes - as in excluding men from spaces where women are vulnerable - it is the lesser of two evils.
And because we've done a good job at removing unnecessary formal sex discrimination, all the sex discrimination we're left with is the necessary stuff, and that's what transwomen are trying to breach.