Exactly. Male people with gender dysphoria don't have anything in common with female people without gender dysphoria (other than being human, of course). They certainly don't have any kind of shared identity which justifies them sharing the same single sex spaces or competing in the same sporting categories.
This discussion is, of course, somewhat outside the scope of what Dr Cass was commissioned to report on, which is how to treat gender distressed children. But I do think it is nevertheless somewhat implicit in her findings. In her report, Dr Cass acknowledges that for some people, whose dysphoria is extreme and persistent, medically transitioning will be the right option. But she cautions against taking any irreversible steps during childhood, or even steps which we might think to be reversible, such as socially transitioning a child by changing their name and pronouns. Why would you need to caution against this if being a trans adult were completely unproblematic and trans adults could expect to enjoy as fulfilling a life as anyone else?
And there's the rub. People can socially and medically transition but they will never, ever be the opposite sex and they will never, ever be accepted as such. Even if you move exclusively in the kind of circles where everyone uses your preferred pronouns and insists that you are what you say you are, there will always be limits. There will always be people who object to you using public toilets in accordance with what you see as your acquired gender. You will never be able to compete in sport without controversy. Any hospital admission could cause issues. And you will find that straight men and lesbians might be willing to say you are a woman but most of them are not actually willing to date you. And the ones who are willing to date you may not want to have a long term relationship with you because they want children, something you can never give them (unless you have not medically transitioned and are willing to use your body to perform its reproductive capabilities in accordance with your biological sex, something which triggers your dysphoria).
This isn't to say that trans people can't live happy and fulfilling lives, but trans adults face struggles and challenges that the rest of us don't face (and that's before you even address any health problems they may suffer as a result of medically transitioning) and so it's not the ideal outcome if it can be avoided.
I think this is implicit in the Cass review, where Dr Cass essentially green lights medical transition in a small number of adults as a last resort where all other treatment options have failed, but suggests that otherwise it's not a good outcome. She says so very subtly, but the message is there. And it's essentially the same thing Helen Joyce said, much more bluntly, about needing to keep the number of people who have transitioned to an absolute minimum. Dr Cass can, of course, never say this explicitly, even in a Mumsnet Q&A.