How about this - a proposition someone cleverer than me made:
If you are going to choose to modify your body so much that it makes people around you uncomfortable, either by having devil horns implanted in your forehead, or by being a woman but taking so much testosterone that you look enough like a man that you intimidate other women, respectfully, that is a you problem. You need to deal with the consequences of your choice.
And one consequence, if you are a transman, will (or should) be that the onus is on you to be aware that you are biologically a woman and therefore you are included when signs say “women.”
Interestingly, once upon a time, one of the criteria for being eligible for “sex reassignment” surgery was that you had to acknowledge that you knew that it would not actually change your sex. And that the world would not actually treat you as the opposite sex. People who couldn’t acknowledge that were not considered good candidates for the surgery, because they were seen as delusional.