But then I wondered do you get female transvestites? Do women who dress as men as a thing exist? Do they have false beards, shoulder padding etc like male transvestites wear padded bras? Sorry if this is an insensitive question, it's just when I was explaining it I was careful to use the word "person" and then I suddenly wondered well, wait...
I've been to a black tie do in black tie. I don't identify as a transvestite, but I did identify as someone who couldn't be arsed to fork out on a piece of clothing she'd probably only wear once. People kept telling me I looked awesome. I was totally aware that if a man had turned up in a ballgown, he might not even have been let in, let alone had a positive response from everyone, even if he looked totally stunning. It's a lot more acceptable for women to want to dress in men's clothing (because anyone in their right mind would want to be a man), but far less acceptable for men to want to dress in women's clothing. And yet clothes are just fabric stuff you put on top of your skin. It's not actually part of you in any way, and it's just social convention. Men can wear dress-like clothes in some circumstances - vicars' cassocks, sarongs, kilts and so on - just depends on the culture. There must have been some women who dressed as men in the past though, because there's a whole history of sumptuary laws around clothing (which also covered status as well as gender), so clothes clearly are important markers, but fashions change, too.
Um, this is just turning into a stream of consciousness.
Meanwhile, back with what's under your clothes - well, mostly (but not always), I have fancied people who look male to me. Actually, it's not just look - there's smell and so on, too. Had a bit of a weird moment in the petrol shop earlier, as the bloke in front of me in the queue smelled like a guy I was with about 15 years ago. Does the smell of people who've transitioned change? I assume that if you take hormones, it must affect what you smell like. Although laundry powder and soap and aftershave and so on also go to make up how people smell. (I am more sensitive to smell at this point in my cycle, so that might be why I've been noticing it and therefore thinking about it today.)
But anyway - under clothes, attraction is nothing like as simple as whether someone has a penis or vagina, or whether they've had surgery or not. Usually that's the least of it - IME, there has to be a lot of attraction on what you see outside the clothes and experience in the personality before you get to finding out what's under the clothes anyway.
I think possibly it just comes down to consent in the end. Just because I usually fancy male humans of a roughly similar height and build and so on, it doesn't mean I fancy all men like that, nor does it mean I've never fancied anyone who doesn't fit that mould. I can't tell you why some people make me feeling ripping their clothes off when others who seem very similar in many ways just don't have that effect on me. Even if I do fancy the pants off someone, I might still decide not to have sex with them for many reasons. I might decide that even if we've had sex together before, or I hope to have sex with them in the future. And even if I do want to have sex, I might not want to do every act that they do. Plus if I want to have sex, it can only happen if they also want to have sex. If either of us says no, the answer is no, whatever the reasons for it, even if those reasons are transphobic or whatever else.
Some of the transactivists don't seem to get this idea of consent. I think an individual's right to say no overrides any of the politics and philosophy around it. Not that that means we shouldn't discuss the politics and philosophy, but I would prefer it to be discussion rather than dictated dogma.