For me it's opening a book and discovering it's written in the second person. I can't bring myself to read on. It makes me squirm too much. Just about acceptable in a (very) short story, but not a whole novel.
I'm also no fan of epistolary form, if I'm perfectly honest.