You are, either through ignorance or malice, combining two different things.
It is disingenuous to pretend the issue with trans people is how they dress. Where there is an issue about dress the issue is the reason they dress this way and the access they claim because of that.
How someone chooses to dress is, as you say, no one else's business as long as it's not indecent for the context like extreme fetishwear in Tescos or something offensive like a T shirt with a sexist slogan.
Conversely, a person of one sex claiming to be the other sex, or the right to be treated as interchangeable with the opposite sex, because of how they think is unacceptable sexism regardless of how they dress.
It's the difference between a man buying a fabulous women's dress because there's nothing like that for men in the market today and having the dress altered to fit his male body, and a man buying a fabulous woman's dress because he thinks he is a fabulous woman, and altering or agumemting his male body to fit the dress.
I think it's interesting that you rightly see the connection between social racism today and the abhorrent legally sanctioned abuses of black people in the past, but are blind to the connection between the abhorrent legally sanctioned abuses of female people in the past (and indeed in many places still today) and the social sexism that women (in the sex based meaning) still face and which among other things underpins the concept of gender identities.
You say "All trans people are doing is getting on with their lives whilst being trans" but what does actually mean? In what way, and why, is someone of one sex reasonably interchangeable with people of the opposite sex because of something in their mind, and how in the case of a trans woman does that quality relate to the existence of female people, our undeniable history of oppression because of our physical sex, and the existence of women-only resources, provisions and protections to mitigate that history such that there can be no reasonable case to differentiate between us?
To get back to the history of race oppression, would you consider it reasonable for a white person to claim something in the way they think makes them really a black person, and through that claim the history of black oppression as their own even though they not only would not have suffered any of it at the time, but as a white person would have benefitted from it and continues to benefit from its legacy today?