Surely, even if you believe you "pass" as a woman, if you start using men's toilets now men will understand that you are a trans woman and are using the men's toilets in accordance with the law?
It sounds like you've never used men's toilets, so what reason do you have to believe that you will be unsafe in them?
In practice I think most organisations will take the view that people like you shouldn't have to use men's toilets and will make an effort to provide alternative individual or mixed sex facilities instead.
Even this may compromise women's safety to a certain extent, because if mixed sex facilities become more widespread, we will be less safe in those than we would have been in single sex ones.
I don't think your right to privacy trumps our sex based rights though.
You do have to accept that the world doesn't revolve around you. There is no way round that anymore.