I think this is a big part of it.
Men, whether they admit it or not, tend to think man = penis and penis = man. (And, to be fair, from the gender critical feminist point of view, they're not wrong about that part.)
So if a man has his penis removed, he isn't a man anymore, and since there are only two types of humans that would make him the non penis having kind, i.e. the woman kind.
I think, as you suggest, they probably also think having your penis amputated is the worst thing imaginable and that anyone choosing to do so must be seriously mentally disturbed but is probably harmless to others, so just let them go in the women's changing rooms if that makes them feel better.
But a lot of those men, when they learn that most trans women keep their penises, then go, "WTF, that's just a dude in a dress!"
And they don't like the idea of someone else's penis being next to their wife/girlfriend/sister/mother/daughter when she is getting changed.
Whereas women tend to be more like, "We're all complex, there's so much more to us than just having uteruses and vaginas, it's so reductive to define a woman by her body parts, it's anti-feminist to reduce a woman to her ability to bear children", but then get stuck on what a woman actually is instead.
That, coupled with the emotional blackmail from the trans community saying they "just want to pee" and dropping ominous hints about suicide into the conversation, makes those women push their own feelings about maybe not wanting to get changed next to a stranger's penis really deep down, and insist that trans women are women and we should all be kind.