It's a combination of ignorance (assuming you cannot possibly be a blood relative given the differences in appearance) and unconscious bias with a racist undercurrent (assuming that you must be the "inferior" paid help).
Contrary to popular belief (or perhaps just that of the MNers on here looking to downplay it by any means necessary), all because a white person may receive a similar comment does not make it not racist when such a comment is addressed to the OP.
It happened to me all the time when my (much paler than me) DC were younger. It was infuriating and, when it happened often enough in a single day, distressing. After the first few incidents, I did my utmost to pull people up on it.
"Are you their nanny?"
"No, I'm their mother."
"But you don't look like their mother."
"But why would that make you think I look like their nanny?"
"Well, because they don't look like you."
"That's funny - I just remembered that I have a lot of family and friends that don't look like me. I'm clearly losing out on a fat paycheck(!)"
I classed it as similar to people expressing surprise at my career:
"What do you do?"
"I'm an (e.g.) investment banker."
"Oh, you don't look like an investment banker!"
"Really? Wow! Please tell me what an investment banker looks like?"
And also at how I sound when I speak:
"Oh, you're very well spoken!"
"Really? Compared to whom?"
I would be told, they don't mean anything by it, they're just trying to compliment you, so just say thank you. Well, if that really is the case, they need to keep their backhanded compliments to themselves 
When recounting these tales, people would make all kinds of excuses to encourage me to brush off the comments as silly/unimportant or dub me "the angry black woman looking for any excuse to be offended", but I disagreed then and still disagree now. I think it's important to challenge ignorance, unconscious bias and racism where you see it.
Saddened to hear that the same shit is still happening to the OP 