Those reactions were a bit rough! I think it's a perfectly reasonable thing to talk about.
I felt the same about reading with my DS, particularly when I saw friend's girls of the same age sitting and practically reading the books themselves while DS found the mirrors in the books and licked his reflection! (Clearly the intellectual of the group!)
Just so you know not to worry too much - he wasn't interested in any book without pop-ups until about 18 months. We basically stuck to 3 or 4 that he really liked and went to the library every couple of weeks, borrowing lots of books, not worrying that he just bashed those while looking at his favourites, until he suddenly developed an interest. We still go to the library every 2 weeks and now I struggle to get him to put the books away to go to sleep.
I understand where you're coming from - I really enjoy reading with him now and it was so nice when he suddenly decided he loved it. (Having said that, he's now 26months and only wants to read books about motorbikes and guitars - oh, help me!)
My mum struggled to get my brother interested in books until he was about 6, he was just far too busy for it. It eventually became their 'thing' to do together, far more than with me. Now he is one of the most avid readers I know.
Maybe it's a boy thing? Maybe it's just our boys! Keep reading!