Oh come on, "Many" is a huge stretch.
Even the link YOU linked to estimated 20-40,000 almost all visiting temporarily over a period of 250 YEARS! That works out as approximately 80-160 people a year out of a population of about 12 million! Or, to put it another way, 0.00001% of the population. And, again, as YOUR OWN link specifically says, they would have been primarily based in London, not rural Hertfordshire!
I've got no issue with colour blind casting as in bridgerton, when they aren't making any professions of historical authenticity and the music, heavy make up, language etc is also a stylistic "interpretation" of the period.
I'm also fully in support of diverse casts in setting where that might be authentic - harlots was a potential example, as would anything set, for example, in the London shipping industry/piracy at that time, or a young rake doing his European tour, or the abolition movement
(those are just vaguely regency examples, obviously the scope is huge throughout history, I'm certainly not saying Britain was exclusively white until 1950).
But let's not pretend that people of colour, particularly in JA's very set social milieu of rural gentry, was common, to the point where it wouldn't be commented on, because it's just not true.