It depends on how they are managed more than the sex separation for me.
In a local shopping centre the unisex loos are immaculate. Staff are constantly in and out cleaning and freshening, they feel safe and the shared area is open and spacious.
By contrast, the women's loos at Victoria station are disgusting and you have to queue in a humid stench pit for the pleasure of using them.
I've come across a lovely set-up in a bar where there is a big open space with a huge circular fountain for handwashing and then two (ungendered) rows of cubicles curving around the wall. You don't have to stand next to anyone to wash your hands, as there is so much space around the edge of the fountain, and the water is operated by a foot pedal, so no one is touching taps with their hands.
I'd much rather this arrangement than a dingy women's toilet where a random person could follow you in, and no one would know.
So, no unisex toilets don't bother me. What bothers me are badly-designed, unkempt, badly-lit, uncleanly toilets.