I think it is a difficult one, with several layers. There is definitely a growing sentiment of anti-Semitism being seen as acceptable within certain circles in the Left, especially during the Corbyn years, and that's awful.
However, David Baddiel is also the guy who did Blackface, so maybe he's not the right messenger for this one. Also, it's hard to see Jews as a distinct race, as to most undiscerning eyes, they could easily pass for White, unless displaying obvious symbols of their Faith.
And finally, sometimes there seems to be an extra- sensitivity around everyday language that's immediately leapt on as evidence of "anti-Semitism", even when it isn't or isn't intended that way, especially concerning the notion of Rich Jews.
Obviously, the Holocaust was a terrible thing, and it's also the only form of discrimination where to even deny it, is a crime. You don't get that for anti-Black racism, for sure.
And then there's the whole thing about someone being culturally Jewish, but not necessarily an adherent to the Faith, but also being extremely sensitive about all sorts of things "perceived" as coded language.