I know women have a much bigger problem with it as we're a larger majority, but it's a big problem in the relatively small trans population
Depends where you're talking about. It's not particularly in the UK, it is in a number of countries in the Americas.
As for who trumps who, unfortunately sometimes that's exactly what it comes down to. Most people would accept that trans women and gender non conforming males are at risk using men's toilets, and other sex segregated facilities such as changing rooms. And that the risk to them will be greatly reduced if they're with the women. It's also the case that allowing entirely free access to women's single sex facilities to any penis holder who wants to come in and claims to identify as a woman puts women at risk, because it's begging to be abused by, well, abusers.
We know that both of these things are the case because there are trans women and GNC males who have been attacked in male facilities and there are males who have attacked females in women's facilities.