Exactly.
Imagine having your annual review "well done, Egg, youve only had one formal complaint for sexual harrassment this year, the same as <checks notes> last year and the year before, good-o, seems like things are going really well for you here...."
He is trying to minimise what he knows is unacceptable behaviour.
It was only 13 and they were only the miserable menopausal women so their opinion can't be trusted!
Also (and this is possibly petty) but saying "a handful" of people makes it seem fewer. I would say a handful of people would be single figures ....not tht it would be ok if "only" 9 people complained but another example of trying to minimise the situation