I actually feel more irritation with non-binary people than trans folk at times.
At least with trans folk, there is a genuinely complicated situation - ie/women don't want trans women in their single sex space because they are biological men, but trans women are worried about their own safety and the threat from men if they use the male toilet. While I don't want TW in female toilets, I appreciate there needs to be a solution (ie/third space).
Non-binary is just a rejection of a social construct. Lots of women reject stereotyping or feel uncomfortable with being visibly feminine. I don't see any reason at all why someone who is non-binary can't use the toilets that align to their biological sex.
I've got quite a number of TW on my Twitter and they're actually bloody marvellous. They believe in single sex spaces and accept they don't belong, and they genuinely raise up women's voices. It's just the vocal TRAs which fuck things up by trying to demand women's rights are trampled on.
But NB individuals....they're frequently younger adults who are clearly still in that stage of trying to figure out their identity. Not always, but often. And they wear it like a badge, inserting the fact into conversations wherever possible. It's just wearisome and I don't think it comes from the same place as genuine gender dysmorphia. I think it's trendy and seen as cool to be "different". Except it's not different because every other bastard is claiming to be NB now. It's a rejection of societal standards and stereotyping - just like punk was, back in the day.
Absolutely no reason for NB not to use sexed toilets. There are plenty of women who look quite butch - and until men started bragging about accessing our spaces, it was never an issue.