But life isn't colourblind so why should fictional life be? It's weird to somehow pretend that skin colour isn't relevant and doesn't have any impact on someone's life - its not just the way we look (although that in itself affects the way we go through life and others interact with us) but our family history, our background, our culture.
I think the overall verismilitude of the performance is relevant
E.g. fantasy dramas - if you can have dragons you can have black people! Particularly if it actually provides context to the story - e.g. it makes sense in GOT/HOTD that the Velaryon and dornish characters have different skin tones, it would be strange if one random member of the starks were black or Asian without any explanation.
Same with plays and musicals - you are already suspending disbelief that a cardboard background is actually a city and people are bursting into song so the colour of peoples skin (or even their gender) is all part of that
With Bridgerton etc the music, dialogue and other stylistic choices make it clear its not attempting to be an authentic replication of the early 19th century, and, again, the creators have provided an in-universe explanation of why there are more people of colour.
Then of course there are all the historical dramas where multiple cultures are historically accurate- not the one in a million chance of having a black bishop in medieval England but ancient rome, egypt, 19th century new york, whatever...
So there are lots of ways casting can be more diverse, without even mentioning the most obvious - historical dramas from/set in countries with a majority non white population (which would be great similarly because its something different - do we really need yet another Jane austen or dickens drama? When we can have a shogun)
But if you are trying to present as historically accurate a film as possible then its doing a completely disservice to everyone - the actor, the audience and the real people who lived in that time to just cast a Asian actor and pretend nobody will nonotice, when really their race would have informed everything about their life. Its actually more racist to just whitewash history and pretend racism never existed as it suggests people were making a big fuss about nothing.
And of course if you are representing a real person they should be of the same physical appearance (unless an obvious stylistic choices e.g. cate blanchett as one of the many bob dylans within the same film). That's not just skin colour but everything else, including gender. If you cast a man as Elizabeth 1 it underrmines the struggles she faced as a ruling queen. A 30 year old Anne Frank would negate the veracity and tragedy of the story of a girl growing into adulthood. A Black Anne bolyen would have had a completely different life experience than a white one.