Ms Badenoch becomes the most senior minister to call the alleged comments racist - going further than the official government response.
Downing Street have called Mr Hester's alleged comments "unacceptable" but without specifying why.
Asked repeatedly if Rishi Sunak would describe the words as racist, the prime minister's spokeswoman said she would not get into "further characterisation".
In a social media post, Ms Badenoch, who also serves as the equalities minister, said: "Hester's 2019 comments, as reported, were racist.
"I welcome his apology. Abbott and I disagree on a lot. But the idea of linking criticism of her, to being a black woman is appalling.
"It's never acceptable to conflate someone's views with the colour of their skin."
Earlier, health minister Maria Caulfield told the BBC she considered the comments to be racist and that she would not accept a donation from Mr Hester "if he made those comments".
https://www.bbc.co.uk/news/uk-politics-68548389
Who would turn down ten million let alone return it after your have spent it? Not of course saying that anyone has allowed the amount of money their response to the alleged comments.