I think it would be a shame not to give him a chance if you really like him in other ways. But maybe, in general, YANBU. I certainly don't think you're unusual.
I have never been in a relationship. Food isn't the only reason but it's a big one. I became anorexic at 11 and still (at 30!) have major 'issues' (hover just outside the official 'anorexic' zone) with most foods. I eat the same for breakfast and lunch every day and there isn't much variation in my dinners either. I don't eat any single item that contains more than 300 calories on its own, I don't eat things with sauces or that have many ingredients mixed together, don't eat red meat, nuts, cream, tomatoes, onions, pastry, Indian/Chinese/Thai/Mexican/Italian, cheese, chocolate, milk, cake, biscuits etc etc. As somebody said upthread it's probably easier to list what I do eat.
I can't ever go to someone's house to eat and I will only go to a restaurant with close friends who I have known for a long time.
I can see why people link food issues to other issues too - I also sleep poorly, enjoy rigid routine and hate both physical and emotional intimacy
However, I am able to travel all over the world and go about life without worrying because I'm not just fussy, I'm fussy and eating disordered; meaning, if there's nothing I feel comfortable eating, I'm very comfortable with not eating!
I've accepted myself as I am and don't mind it but I wouldn't expect anyone else to accept me. The only thing I'd change is that I wish my children had a healthier role model.