Yes but the "of" mistake originated from people mis-hearing the contraction or the elision of "would" and "have," which is a common feature of spoken English in many accents.
It is certainly a pedantry too far for people to object to, or worse, correct people for speaking the contraction or the elision.