Everyone knows what sex they are (with very few exceptions generally among children in countries with v limited healthcare).
Single sex spaces used to work because most people knew to go in the one for their own sex. If they didn't, deliberately, they'd be considered a pervert or predator (if male) or confused/inappropriate/potentially at risk (if female). Going in the wrong one accidentally was embarrassing. That's because not only women, but also most men, like privacy from the opposite sex in the toilets (and in various other spaces where you might be undressed, vulnerable etc).
Anyone breaking these rules could be freely challenged or reported and security or police called if they wouldn't leave. That stigma and risk meant even most dodgy/predatory men wouldn't bother.
Now, according to TRAs and many deluded organisations (though thankfully not UK law) we're meant to celebrate and welcome men with fetishes about the idea of being a woman, and even worse, recite the lie that they are women and have them in all our spaces, just because the man says so! If you can't see that a bunch of the worst predatory, fetishistic and pervy men are going to grab that opportunity with both hands and use it for sexual gratification, for a sense of power over women, and for the enjoyment of making women uncomfortable, you must be very dim indeed.
And it's not just about "attacks". There most certainly have been attacks on women and girls by trans-IDing males in women's spaces, including toilets, hospital wards and prisons. But it's much more than that. Ogling, upskirting,, taking photos over or under stalls, verbal harassment, intimidation - they are all reasons women should have single sex, actual woman only spaces. They all make women and girls miserable, scared and in many cases self-exclude to avoid them. There's also the issue of women whose religion or culture means they can't share such a space with males, so they self-exclude. These are often the ethnic minority women you like to use for your argument.
Finally is we have self-ID and any man who just says "I'm a woman" can go in women's spaces, then the men who you think want to attack trans women can also do that, and come in after them. So "inclusive" toilets and other spaces actually have no safety or dignity for anyone.
No it's not always going to work 100%, no rule or law does. There is such a thing as a successful disguise and some people will always break any rule. But we have categories and spaces that exclude or include on the basis of age, disability and other things that can be hard to prove or identify visually immediately. (And that actually are a spectrum too, unlike sex.) They work because the law, institutions and organisations are prepared to enforce them and have reality-based ID for them, and people are happy to report fraudsters, as they haven't been comprehensively lied to about the law and reality for over a decade by activist organisations, and can do so without retribution. Because of this, they work, because most people don't take the piss.
If you want to pretend you're 70 when you're actually 60 to get your pension, or that you are have cerebral palsy when you actually don't to win in the paralympics, that will probably not work out for you - even though you could do a really great imitation, even though it could be hard to tell. That's because most people don't want the stigma of being exposed as a fake, people wouldn't be punished for reporting them, and the systems we have would expose them soon enough.