I think the problem with it coming from the Tories is that the TRAs are trading quite heavily on people’s desire to be liberal, open and diverse, but without understanding that one of the liberal tenets is freedom but that you should not do harm to others. The question of harm has been less well articulated for fear of being branded transphobic. The problem with it coming from Davis, is that his views may just be dismissed as conservative and retrograde, without engaging with the weight of his argument.
And to an extent, there is a conservative element to the idea that biological sex matters, if the person arguing it also believes that there are inherent hierarchies embedded in the sex roles of male/female. However, the sex roles are what we mean by gender, and the feminist view I think is that gender is a social construct, not that different sexes don’t exist and should not be protected.
So the fundamental premise, that different sexes exist and should have certain rights protected (to same sex space), aligns in conservative and feminist thought. It is going beyond that basic premise that viewpoints differ, because feminists would see sex as the basis of oppression, whereas conservatives would see it as a natural order. I think.