People can be really bitchy about this kind of stuff. But it often says more about them than it does you, IMO.
Some people really do seem to believe that if you're, say, a solid seven then you have no business going for a nine. And while it's generally true that people do tend to stick closer to their own perceived levels of attractiveness, just bear in mind what this is actually based on.
The bones in someone's face. The muscles on those bones. Skin. All just a load of organic matter that's no deeper than the tip of your thumb, and entirely arbitrary at that. And your friend thinks this determines a "batting average"? She's limiting herself. But so what? Less competition for you, right? 
(yes she's married, but YKWIM)
Looks alone are nice and necessary, but say nothing about a person's personality. You'll know attractive people who are brainless and dull, as well as those less easy on the eye who are engaging and interesting. Good-looking guys who are a little bit uptight so aren't really that sexy, but some guys who just have an ease with themselves that radiates grrr. The personality fills in the canvas of the face, no?
Fact is, you clicked with this guy. And that's not just looks-based. You obviously have something about you that pinged his radar. Confidence? Chemistry? Who cares? If you've got it, work it!
Looks might give you a bigger net. But then so does just having a bigger net 