Also @DadJoke, I'm not sure you're aware of this, but trans women aren't the equivalent of black and gay people in your analogy.
When black people weren't allowed to use the same public toilets as white people in parts of the US in the 1960s, for example, the black people were the oppressed group being excluded from public spaces by the group of oppressors for reasons which had nothing to do with safety, dignity or privacy.
We don't have single sex toilets for men and women in order to oppress men by treating them as lesser humans. We have single sex toilets in order to protect female people (the oppressed group) from male people (the group of oppressors), as well as for the dignity and privacy of both sexes.
By allowing male people into single sex spaces for female people on the basis that some of them believe they are not like other male people, you are removing that protection from female people, who are the actual oppressed group here.
If you believe that women aren't an oppressed group and that single sex spaces are merely evidence of bigotry towards members of the male sex, logically you should be campaigning for the abolition of all single sex spaces. But you aren't doing that, and neither are trans rights lobby groups. Indeed, they insist that trans women must be segregated from men.
If you believe that women and trans people are both oppressed groups, logically you should be campaigning for both those oppressed groups to have their own dedicated spaces (i.e. single sex spaces for women and additional spaces for trans people). But you aren't doing that either.