I think the issue that causes annoyance isn't the mistake in the first place, but people's willingness to change and learn from their mistakes.
Several people in this thread have asked what the issue is, and my response to that is where do you draw the line? There are some mistakes that are obvious and are therefore quite easy to understand assuming English is your first language. If it is not your first language, or it's a less common mistake, the communication is more difficult to understand - and that's where the problems start.
The whole point of having a common language is so that we can communicate our points/needs/information as effectively as possible. The further people stray from 'proper English', the more difficult it is. Therefore, we have to try to be as correct as possible, or we'll end up with complete sub-languages.
I have worked as a proofreader, writer and editor since 2012, and have seen some bizarre mistakes in the documents I've worked on. Rather than just correct things for the authors, I explain why they're wrong, and will tell them if it's a common mistake. They usually thank me for this, and some have been amazed at words and phrases they've always used incorrectly.
I am certainly not infallible, either. Until I was about 18, I would say 'seen as' instead of 'seeing as' and, like many other examples in this thread, it's due to the way people speak where I was brought up. You write what you hear, and that's fine. But my dad corrected me on this one, and I was amazed (and quite excited to have learned a new thing, if I'm honest), and I've never made the same mistake again.
That's what it's about.
Quite obviously nobody is having a go at those with obvious learning difficulties like some have commented on. Mere those people who are simply unwilling to take note of the things around them and notice when they've made an error.
There are lots of people with learning difficulties, or dyslexia, or whatever - but there are many, many more who simply cannot be bothered to communicate properly, and that's the annoyance.
It's similar to a thread about being annoyed when people walk dragging their feet, or slouching badly. Yes, there are people who can't help that - but there are lots more who can.
*PS, this hasn't been proofed!