What you are asking is what is the relationship between intention, meaning and responsibility. All three can come apart in different situations.
For example, in Greek the word ‘malakas’ is both an offensive swear word and a term of endearment, the difference comes from intention and context. If you call someone this in anger, intending to insult them, it is insulting, if you call someone this with affection intending to indicate they are your friend, it is a positive word.
Swear words tend to have a negative connotation, they are usually used to insult, but, as above, this is not always the case. If someone uses a swear word in ignorance, by accident, or mistakenly and does so non-culpable, we wouldn’t hold them responsible. For example, a child who uses a swear word they do not understand, either the exact meaning or the usage as a slur, we wouldn’t hold them responsible. Because words change meanings, it is possible for an adult to make a non-culpable mistake in use; here intention is crucial, we don’t hold people responsible for unavoidable mistakes.
The connection between intention and meaning is also evident when words are reclaimed. What was a swear word can be taken over by the community it was intended to offend, embraced and repurposed until the old meaning is changed to a positive one.
I suspect from what you say that your DH made a mistake without intending to be racist and became defensive when you pulled it up on it.