Ah, he always used to come across as a lovely warm person.
I think attacking an individual personally - rather than their opinions - is almost always a bad idea. It's bad because it suggests you're not a very nice person and it also suggests that you are not able to argue your case on its own merits.
Does it matter that Kemi Badenoch is a black woman? Would it have been OK if he'd attacked a white man? Well, no, it wouldn't have been OK if he'd made the same unpleasant remarks about (say) Jeremy Hunt. But yes, it does look worse when it's a Black woman, because of the long historical tradition of men trying to silence women (witch trials etc) and of white people persecuting Black people, through slavery, violence, rape and denying them basic civil rights. That's the context that makes a white man telling a Black woman to shut up particularly grotesque.
It's interesting that an apparently intelligent person like Tennant couldn't see that.