The problem as I see it is not so much what he said, but what he didn’t say.
Saying ‘Men are men and women are women’ is not enough. Those words are literally true, but they are also a dog-whistle for hatred towards trans people.
I believe that men are men and women are women. I believe that nobody can change their sex. I believe that surgery and hormones can help someone with gender identity issues be more comfortable, and that that’s a good thing.
There are however, people who believe that men are men and women are women and who think that anyone who says they’re trans is a mentally ill deviant. They think that such people need correcting, and won’t refer to them as they wish to be referred. They deny trans identity. They feel that children should be taught to think the same way about trans people.
I can’t disagree with what the PM said, because it’s true. But I think he was saying it to appeal to that second group of people. He was, at the very least, not saying which of those two groups he sits in. As a result, that second group of people will feel emboldened. If I was trans I might feel scared by that, and I might wish that the PM had added some nuance.