I don't recall the show being massively funny, but I don't recall it being massively racist either. They tried to take the piss out of everyone in the UK.
I would definitely feel differently if they had only portrayed different races on their show, e.g. if they had only played black characters, as that means they should have cast someone else. But to me, I think it feels different that they played all of the characters - they didn't just play different races, they played different genders too, and didn't seem to mind how ridiculous they looked. So they were also being laughed at themselves.
As much as I don't like being the butt of a joke, I think it feels more comfortable if everyone else is the butt of a joke. Because that's still exclusion. A better kind of exclusion, but still exclusion.
I'm not white, and I have experienced racism, and I do strongly feel that if a part is specifically written for a non-white actor, it should be cast appropriately. (Many parts are more flexible, but if you set a show overseas in the past, you're really going to struggle if you don't consider race.) However, I'm just not sure Little Britain was offensive - not particularly funny, but not offensive.
I might feel differently if I rewatched it now but I can't do that given they've pulled it!