My point, though badly made I admit, is that people often do this to impress. Orwell's 1984 is one of modern days most quoted books but it is also the one people claim to have read when they haven't or they got part way through and gave in. Most people have no idea what it's about yet still happy to churn out quotes from it. This isn't the only book they do this to.
I don't think we can assume what people do and don't do for leisure. I've met many who read widely for pleasure and that includes Ulysses and Crime and Punishment, and before anyone points out Dostoyevsky didn't write Ulysses, yes I know, but it is a complicated book hence the mention.
If you gain pleasure and insight from reading that's great and what makes reading worth it. I don't find reading a pleasure, but I read mainly to gain knowledge with the occasional tap into Christie, Adams, etc. it takes me a very long time to read a book.
I think benefit of reading novels is to introduce you to situations and thoughts you may never ordinarily come across in you life. It doesn't make you fully understand those things but it can help give you a small connection to other worlds.
I just don't think reading in and of itself makes people smarter. But that's ok, as long as people enjoy it.
Oh and because I read so damnably slowly, I do remember whole tracts of books and amazingly people have assumed I'm well read when in reality I have just accidentally read something so much but, the memory lodged it in there. I don't do it to sound well read, but some people definitely do.