In the UK (but not so much in the US it seems) there's a stigma associated with grown-ups reading fantasy books. We're allowed to read thrillers, murder mysteries, literary fiction, anything with vampires, even romance, but if somebody picks up a sword ... that's lame apparently. Magic is for children. It's ok in Harry Potter ... or if it's vampires ... but past that, it's for kids and sad annoraked men on the bus.
Many fantasy books in the UK have subdued and misleading covers so the reader can carry them in public without shame, and the term 'adult fantasy' is more likely to summon images of PVC and whips than of fantasy books written for grown-ups (at work the firewall has blocked off any sites tagged 'young adult fantasy' with the auto-generated explanation 'pornography').
Have you encountered these attitudes? Ever pretended not to own the fantasy book on the arm of your chair :) Are things changing?