Using 'of' instead of 'have' is nonsensical, not just incorrect. It can alter the entire meaning of a sentence. And to many people reading said sentence, it will necessitate stopping, re-reading, identifying the error, mentally replacing with the correct form before reading through again and moving on.
Ensuring that you are clear, correct and easily understood is the basic courtesy of any form of communication.
No-one is suggesting anyone should have perfect grammar, spelling and punctuation. No-one is able to do this all of the time, even with an amazing education and a gift for languages. But I would venture to suggest that should an erros be pointed out politely, those making that error should acknowledge and endeavour to use the correct form in the future.
Because language is important, and standards are important. I'd not seen this as a homophonic error prior to approximately six months ago and yet now it is ubiquitous. That's all the proof you need that allowing errors to go unchecked makes a difference.
It is an extremely basic premise - akin to there/their/they're, or it, its, it's.
YANBU