It's only partly about who he was speaking about, it's also about what he said.
Actors are people of words, so either he can only read from scripts and is otherwise a childish moron, or he chose to say something appalling about another human being.
When people are in positions of power they also have their heads above the parapet all the time. Words are one thing but they also drive others to ridiculous acts of violence, we've had a number of MPs assassinated in the UK over such strongly held beliefs before. In a debate this toxic, it's not beyond the realms of possibility that someone might choose yet another MP to assassinate or harm.
Couple that with Kemi Badenoch being a black woman and we have someone who is very vulnerable. For this MP to get anywhere took a lot more than the usual pathways most MPs take. There will be other people looking for an excuse to target MPs like her too.
So at best he's an idiot who ruined his shiny media persona, at worst he's inciting hatred against someone who is a very easy target purely on the basis that he personally disagreed with her stance and couldn't find better words to make a point other than shut up and to want to live in a world where KM didn't exist.
You don't have to admire people or agree with them, but basic respect for people is due and if you feel strongly about something you need to have respect for that too, all he did fortunately in this case was damage himself and his cause.
So it's not just because KM is an equalities minister, a woman, or black, it's because basic respect, and important good debate went out of the window and potentially made someone who is a target anyway, even more of one all because he can't express himself properly or he purposely chose to make her a target for more hatred than KM already is, that's reprehensible on many levels.