Your point is taken, but let's be realistic here.
If you have never found a black man attractive that does not make you racist. But if you were to go around announcing that you would never date a black man it does rather look like racism, even though you are perfectly at liberty to turn down people for any reason you want.
Given that the difference between your dream date and a trans person is your dream date saying "I'm trans" and then never doing anything about it, in theory surely, we would all willing to date a trans person. Just not a trans person who fights against our rights, or presents in a way that you doesn't match their unchanging sex, etc etc