Words are funny things. Some I just don't like the sound of, others I don't use because they have racist/offensive connotations. It's not just about how a word makes me feel, it's about its impact on other people. If some words are commonly issued in an offensive way, then even if your intention is not offensive, the word still sounds cruel so you shouldn't use it.
Language is very fluid though, as others have pointed, you have to be aware of the evolution of our language and adapt when things change. Queer is a good example of how an adjective went from derogatory to acceptable. Words can be reclaimed and/or fall in/out of fashion.
Re the term coloured, I wouldn't use it anyway because it currently is said with offensive connotations, but I personally hate the term. What I object to is not the notion that some people have a skin tone other than white, but that the only possible alternative skin tone is dark.
People always use coloured to mean black. People always use the term Asian to mean Indian. What about the rest of us with a different shade, eh? I often feel excluded from these ethnicity tick boxes and descriptions and feel invisible. Hate it.