"no one is left out" - except the people who can't use mixed sex facilities for whatever reason, whether religious, trauma, or anything else.
I'd much rather have someone XX next to me, however strong. They are unlikely to attack (female socialisation, rates of violent crime in females etc show this), and if so, it's not likely to be rape. Most men aren't predators, but we don't invite them in regardless, because of the small number who are and the inability to tell in advance which is which.
Problems staying with small children is a different issue; family toilets could possibly solve that.
Of course someone who is XY is no better than someone who is XX, or vice versa, but they are different in terms of their sex and their requirements for single sex facilities.
If mixed sex facilities are the answer, why were women curtailed from public life for so long because of the difficulty finding suitable facilities when out? There is a need - privacy, safety, dignity - for single sex facilities, and most women want to keep them. Campaign for extra toilets if you really want mixed sex ones as well. Or family toilets or whatever else you think is needed. But don't do it by taking away needed facilities that others want.