For some young women, single-sex accommodation is the only option acceptable to them (or their parents). Changing the rules to allow men who identify as women in to the college removes those young women's ability to access the whole university.
Conversely, men who identify as women can access the vast majority of colleges in the university.
If you're balancing up who is most disadvantaged by this change, clearly its the women. I think the only men who lose out are those who want to be at a (previously) women-only college because that validates their gender identity in a way which merely living as / presenting as a woman doesn't do, or doesn't do as strongly.
The debate on where there should be single-sex colleges at all is interesting, and I'd like to see the research / stats on whether it broadens access, whether women tend to do better etc. But this seems to me to be a different issue.