I think a rule of thumb is
a) 'would a (black, gay, religious, insert whatever the joke was about) say it?
b) would it have a slightly different more or less offensive meaning if they did?
And the joke in this context I think could be said by a black person, but I think it does have undertones of racism when said by a white person to other white people.
I'd probably tackle it as "it may not be intended to be racist but it can easily be interpreted this way"
Another example is a friend of mine who told me, "You're not white really, you are one of us" meaning black/Jamaican
I laughed and took it as a complement, but I don't think I would say the opposite to her because it could be interpreted in a different way.
It's like saying Asians are good at math. Sounds like a compliment. Until you remember that Asians are not a monolith either, and some of them are good at math, but some of them aren't, and instead of lumping them all together because they aren't people to you, just fodder for your crappy, uninformed, ignorant jokes and comments
LOL yes I have to frequently 'tell off' one of my class, he is good at the maths/logic stuff, he often says things like, "yea, I got it right, that's 'cos I'm Asian" and I remind him not to stereotype and make comments about race.
And the number of times I have been told that 'half caste isn't racist'