They can't determine it from skin and eye colour together either, as everyone is saying.
There are siblings with the same heritage who define their ethnicity differently. It's not just where your eyes or skin tone comes from, it's also cultural links and perceptions of connection.
My children all have the same heritage, and happen to have the same skin tone & hair colour, very similar eye colour, but my older 2 identify their ethnicity differently. Neither is wrong, they're not picking something that isn't in their heritage, but they see their mix of heritages differently and possibly in a way people who look at them won't be able to guess. My siblings and I all have very different eye colours and skin tones with the same mixed heritage, and we've heard all sorts of guesses in our time, the worst probably being that we're too different to actually be siblings. That fucked with my head a lot when I was kid.
If someone in your daily life is not believing his background then I can see why that would be irritating and would at least want them to not talk about it. With randoms, eh, people are going to make some assumptions based on appearance, and sometimes they're wrong, more often if from a less represented ethnic group. It can feel uncomfortable or awkward particularly if the guesser makes a deal about it or makes it out that you're lying (been there, had that, though thankfully only a couple times), but mostly it's either just a tickbox exercise or small talk and I'd change the topic as quickly as possible.