Some of these things are quite interesting in terms of a development of language element.
There are of course words for groups that are intended to be derogatory, some started that way and some became insults over time.
But words like "coloured" didn't really have that kind of natural evolution at all. It was never derogatory in its meaning. It was literally a matter of, a few people decided it was old fashioned, they wanted a new word that would have no baggage, and they told people to stop using the old word. It was very much based on theories about changing language to change attitudes. And in these cases, it was not picked up by everyone in the group, either - there were significant numbers of people who continued to use the old words for themselves without issue until that generation passed away.
It would be more accurate to say words like that were archaic, or old fashioned, rather than racist or derogatory. Except that wouldn't fit with the underlying purpose of those who push such changes, which seems to be to designate new in and out groups.
Words like "Frogs" or "Krauts" are similar in that they were not actually derogatory in themselves, any more than "limey" is. They were nicknames, typically used by the working classes. (They are also, now, somewhat old fashioned.) Using nicknames and substitution words was until fairly recently(and even now among certain groups, interestingly mainly racially united groups, my partner's wc Caribbean family has a ton of these kinds of terms) a significant component of wc language - including nicknames for individuals. In my father's generation for example, more than half of his friends were referred to by nicknames - some were related to something about the person, maybe an incident that had occured - others were standardized, for example guys with the last name Miller were always called Dusty in our area. You similarly see a lot of affectionate pet names among the wc people (luvie, duckie, etc)
Frankly, I think a lot of the inability to differentiate between racist language and the language traditions of differernt social classes comes down to class elitism, and it's the whole point of pushing ever-changing "correct" language and designating previously inoffensive language (or even correct in the case of coloured) - it's to be able to catch people out who can't keep up.
The people who get worked up about this stuff are typically the same people who are upset when some wc nurse calls them by a pet name when she gives them a shot, or who insist that someone shortening their child's name is deeply offensive. (Noteworthy that wc nicknames are not chosen or approved of by the person being named.) It comes from a middle class sensibility around language that will not in a million years be tolerant about language traditions of the working classes, because so much of being aspirationally middle class requires not being mistaken for that sort of person.