Toilets don't bother me Argy (though I, unlike you it would seem, do accept that other women may not see it this way). However, this issue is not just about toilets.
My local university's pool (open to the public at weekends) however, has open-plan, single sex changing rooms, no cubicles, no curtains. Would I want to see a person of indeterminate gender with their penis hanging out, even if they were changing into a tasteful one-piece and intending to tuck? No, I would not. Likewise, were I to go for a cervical smear expecting a female nurse and be confronted with someone who merely identified as female, I would not be happy about this. If I booked a sleeper berth, and found myself sharing with a be-penised individual, regardless of how that individual identified, I would not feel able to sleep (and that despite having shared mixed sex dorms in mountain huts - the one-on-one nature of a two berth sleeper would make me feel much more vulnerable).
You may be fine with all these situations, and if so, good on you, crack on and live your life how you choose. I however am not, and my boundaries and desire for single sex spaces where appropriate are every bit as important as your desire to signal that you don't give a shit, or (some) trans-identifying individual's desire to force the rest of the world to treat them as the sex they would like to be in all and every conceivable set of circumstances.