I wouldn't be scared of being called racist. I find it silly that people are so quick to be offended by being called racist.
I am sure that in the course of my life I have sometimes been incompetent, selfish, ignorant, thoughtless, uncommitted, unprofessional and inappropriate. We all are sometimes. Sometimes people have pointed this out to me. Of course I do not want to be criticised all of the time, but sometimes appropriate people like a close family member or a senior person at work will point out how I could change. This is helpful to me as it makes to develop and become a more capable person. Sometimes I feel that the criticism is unwarranted and I choose not to change.
In much the same way, I have sometimes made remarks that others may view as sexist/racist/homophobic or distressing to people with a disability and had it pointed out to me. Sometimes I feel that criticism is unwarranted and I ignore it, and sometimes I realise it would benefit others and myself if I changed that attitude or way of speaking.
I am distrustful of people who never wish to be criticised for anything. As for people who are prepared to accept criticism unless it applies to racism - I don't understand that. Is it because they don't believe racism matters? Or is it because they believe they are the authority on the subject?
I am sure we can all think of examples of ill-informed political correctness. We can all probably think of examples where we have seen someone be criticised for being rude. Yet nobody suggests that contemporary society is too polite.