Okay, so trans identifying folks say that they need access to women's spaces because they aren't safe in men's spaces.
This is just one of many reasons the TWAW brigade give though.
This one is easily refutable for the reasons you state and on the basis that it always falls to women to solve men's problems and to move over and make space for them as we are essentially second-class citizens in society.
Most - if not all - reasonable people believe that both transwomen and women should be able to go about their lives safely and free from discrimination.
So while it would cause some logistical upheaval for businesses, if it were merely about trans safety issues, this problem could be resolved relatively simply using third spaces (in a similar way to how people originally said there was no room for disabled toilets or wheelchair ramps, but these are far more commonplace now despite the fact it apparently 'couldn't be done').
As an aside, for me personally, it was the safety issue cited by transwomen a decade or so ago (before the narrative shifted to TWAW) that prompted me (and many of my contemporaries) to be supportive and accepting of transwomen in women's spaces, as we knew only too well how risky men's spaces can be. So it's an extremely effective argument to garner women's support.
But third spaces obviously aren't acceptable to people who believe TWAW, as they don't believe that third spaces would be separating biological males from biological females; rather, they believe they would be separating two types of biological females (one type whose 'inner gender' aligns with their physical body, and the other type of biological women whose bodies were born male) and relegating one group of these actual women to second-class women by pointing out their differences and making them use separate facilities.
So the while safety is often cited as a concern, the crux of the issue, IMO, arises from the notion that transwomen actually are women, in all material respects. This is what causes the issue, because if one accepts that TWAW, then one would be discriminating against women by excluding transwomen.
The conflation of gender and sex over the past decade, along with the GRA introducing the concept of 'legal sex' has meant that the everyday and legal definitions of "sex" have been expanded to interpret transwomen as being of the female sex, in at least some instances. And the fact the law says this can be possible in some instances opened the door to the fact it is actually possible to change one's sex.
So while people might say transwomen should be in women's spaces because 'men are dangerous', 'transphobia can be deadly', 'why would being trans make someone more dangerous?', 'you're scared of little old me?', etc, the only people who will always stand firm on transwomen needing to access women's spaces specifically (aside from those with nefarious designs) will be those who firmly believe that TW actually AW.