Toilets at home are certainly unisex, but the people using them are all family or invited guests, not randoms!
What part of the difference do the people making this comparison not get? Are they genuinely stupid or cynically disingenuous?
If a busload of unknowns stopped at my door and wanted to queue for my loo, I'd soon send them packing.