What a strange remark. Of course you'd know the difference is there was any question of a sexual relationship.
It would never get to that level. The few transmen I've known, while to me passed, weren't people that I was attracted to. Just like many biological men that I'm not attracted to for various other reasons. Don't even consider them in a sexual way, so don't know why you find that odd.
If I met someone and they were upfront and told me they were a trans man, I wouldn't be attracted to them.
That's one issue. The other issue is, do I consider TM men. Yes, simply because they look like men to me so I wouldn't know the difference.
I've never had an issue (yet) of being sexually attracted to a TM because I suppose 1) I know very few and 2) I don't find them attractive.