How long have you known him? Has he made it clear that he fancies you? It's not too picky to expect to fancy someone but sometimes the men you are instantly attracted to are the overconfident/arrogant types who turn out to be total douchebags.
Going on one date isn't leading someone on. If you usually fall for arrogant twatty douchebags then maybe you just don't fancy him because he's not what you are used to? Sometimes you can get stuck in a rut of always going for the same type because it feels familiar.
I definitely don't think you need to worry about leading someone on. You could just go somewhere casual for a coffee.
OP i dont think its shallow at all but who knows he might grow on you i think i speak for most women when i say most of our exs are as Akiss... puts it twatty douchebags. some good friends do become lovers ;)