It's what the person is and not being used in a derogatory manner. My colleague felt he had to whisper though when describing another colleague to me.
I fail to see how describing someone by their ethnicity is any different to describing them by their gender.