You are female. Use the women's loos. It should be that easy.
But, as we all know, transmen tend to have a lot more medical and surgical interventions, go to vocal coaches etc, do all they can to appear more masculine, male. And they often 'pass' very convincingly. So they would not be welcomed in a female facility, they would trigger the same thoughts and feelings as a man would.
I see this at work. We asked our client base about a transman joining the groups and there was a lot of discussion. One transman joined the group and left quickly. The visible recoil from the other women in the session was visible, the anxiety levels palpable and, even after apologies, nobody was comfortable. So he went off to another session, where the staff have specific training in supporting trans individuals.
As far as I can work through in my head, this is the obvious consequence of 'othering' yourself so much. Sometimes you find yourself on the outside more than you wanted.
But as you @Whatiswrongwithmyknee sound as though you are not trans, simply not gender conforming (like so many of us), use female facilities. Unless you choose to mask yourself your voice, general demeanour, gait etc, would not say 'male' and any mistakes would be easily remedied.