If I walked into the women's loo and found a man in there I'd freeze then make a hasty exit. So no, I wouldn't be comfortable with it.
Possibly they might be women who have experienced male violence at some point in their life
Well yes I have, but that's not why I'd react that way, I would have done so even before that experience.
If Andy Burnham's being urged to tell us his current stance on this issue, and he's hedging, then you can be sure he doesn't want to lose the trans vote, which maybe to him seems more important that sticking with the actual biological reality!