I'm so sorry, OP! 
Frankly, I never gave too much thought to the sex vs. gender question until something like this happened to a friend I've known and loved since we were fighting over shovels and buckets in a sandbox (and were both much too young to worry about romance, sexual attraction or the politics of either). Rather accidentally, I grew up to think blokes were the shit, though, whereas she grew up to think it was women all the way.
Hence: no, you're not wrong! Far from it!
While I'd consider myself a "arguably GC but otnerwise more liberal than radical feminist" for all practical intents and purposes, this is where I'd most definitely draw the line: we all - and I literally mean "all", male, female, race, class, education, political leanings and faith aside, even though I've opinionsbon allnof these - get to say who we are and are not attracted to. And, in my book, there is no racism, sexism, homophobia, transphobia, socioeconomic superiority or inferiority complex or any other, however random, situation that's not legitimate in this particular context.
For what it's worth: I'm a white (fair skinned, blonde and blue-eyed), upper MC, heterosexual woman. I like my men stereotypically tall, dark and handsome. And I'll throw "brainy" into the mix.
I've dated or have pined after short men, fair men, not stereotypically "handsome" men in my life. But this is down to me and my choice!
Trying to impose rules for whonwe should or should not be attracted to? No! That's several steps too far!
Now, I've never personally felt the urge to shag a short, blonde swarty female Brexiteer. And I suspect it's rather unlikely that I ever will.l, nor am Iblikely to get itm But if that's what you want: more power to you!
Anything else is nothing short of incredibly rapey!!!