There is a reasonable expectation from other people that a female can get pregnant and a man can't.
We can tell the difference between men and women even if someone decides to transition. Women seem to be able to do this even if men sometimes struggle - it's believed to be a survival type instinct.
English law has this principle through it about what an average person will reasonably believe. If an average woman can identify most males regardless of whether they have a certificate to the contrary and an average man can identify which sex they wish to rape then the reasonableness test about whether they think they may or may not have the capacity to get pregnant remains. Individual health conditions are effectively outliers which are irrelevant to the concept.
A male going into a female toilet has a reasonable belief that all other people they will encounter will be female rather than male and that all of those under 50 will have the capacity to get pregnant - even if they actually are infertile. All women over 50 in that toilet potentially have the capacity to have been pregnant and have female reproduction systems (and all the experiences that go with that).
This is why socially women in their thirties are asked repeatedly "do you have children?" Or "do you want kids?" Which can cause distress or annoyance to women who are either infertile or do not wish to have children because the social currency of their value and position in society is often judged upon their child bearing status. Men are rarely asked the same question in the same manner or with the same frequency. Because we recognise the reproductive ability and value of females and don't give much of a shit about the reproduction ability and value of males.
By the same token, males who transition never understand this social status and system of value. They are recognised as males by polite society and therefore despite 'being treated as women', they are treated as males with the same social reproductive value and status as any other male. This applies to the polite society question of 'do you want kids?' as people recognise this is probably a touchy / difficult subject (women are not afforded this courtesy).
As someone who has to navigate the polite society question of "Do you have and brothers or sisters?" and well understands that "Do you have kids?" I understand the two questions fall into this category of 'Safe and uncontroversial questions you can ask the person sat next to you at a wedding'. They are regarded as icebreakers and investigative questions to help one individual relate and identify with another individual. They are about safe ways to find common life experiences and shared experiences.
If you were sat next to a transwoman at a wedding, you would not ask them about their reproductive status. It would be overstepping and impolite. You would avoid the question. You would know it would be problematic and go into that ground of topic to largely avoid with people you didn't know extremely well first.
This is where transwomen really really don't get that women have shared identity and life experiences that they will never ever touch because of the polite society filter. There are many subjects and topics which will be avoided deliberately in their presence for various reasons. And these reasons are precisely why women want single sex spaces so they can speak about them freely without fear (which includes fear of upsetting others as much as fear of men or fear of embarrassment). These are topics which are irrelevant to males because they don't have a female reproductive system OR the same social pressure of the expectation that they have a functional female reproductive system.
It is about the status, expectation and value of females to our society as the sex that has the ability to reproduce, not so much about the ability of individual females to carry out that role. A man who transitions has no place within this paradigm. Which they know and everyone else knows. And ultimately is the thing that drives them insane and is what they seek to erase and destroy for women.
Validation as women requires the erasure of the visibility of women as the reproductive humans which have reproductive experiences and expectations (which also covers female infertility, complications with childbirth, health conditions related to the reproductive system and the potential consequences of rape).
This is also why we don't see the removal of the word 'man' from NHS literature. It always comes back to the reminder and the concept of 'polite society' and reproduction. Note here too that 'Be Kind' stems from the polite society class. And that women of a certain age tend to get pissed off with the tedium and constraint of polite society (usually because they recognise by this age that the concept of polite society doesn't tend to always work for women's interests by the time they hit their mid 40s so decide to abandon it because it doesn't serve a useful purpose to them and only serves to treat them as service humans to the desires of male interests).
Men don't understand how constrained female reproductive ability puts them within a social context as well as a biological one.
(That was a rather longer explanation than I'd intended).
For erasure of this ongoing conversation about reproductive ability also see discussion of transwomen going through 'female puberty' and 'never having had a male experience' and whom don't like it when it's pointed out that they were once little boys who had willies and pissed standing up and that it's very male to have ever possessed a willy (and again it's an acknowledgement of a lack of a female reproductive system). You also have fetishisation of periods / menopause which we've seen numerous manifestations of. These are not merely odd. They serve an intrinsic purpose for the males doing it, which always comes at actual cost to women who have potential reproductive capacity (which includes infertility).