^^Very much this.
What Brand did was abhorrent but there have been whispers about him being a thoroughly unpleasant man for many years. In his own autobiography he admits to, and details, vile things that he's said and done. We just didn't know the full extent of his behaviour, which turns out to be far worse than we all suspected.
You don't have to like Noel Fielding's humour. He's always had slightly quirky, offbeat humour that wasn't mainstream so I thought he was an odd pick for Bake Off. But anecdotally, he's always seemed like a nice person. There have never been any suggestions about lecherous behaviour and people who have met him in real life have always described him in positive terms. He often pops up on the "what nice celebrities have you met in real life" type threads.
I don't think there's anything to cancel him for. You don't have to like him, but that's not the same thing.
I've seen NF on lots of shows like Buzzcocks etc and he doesn't normally make rape jokes etc - that's just not his usual humour. His jokes are usually a bit weirder - think The Mighty Boosh, a show that he co-wrote.
But if he did make a rape joke when paired with Brand, I'm not going to cancel him based on that. Writing off a person on the basis of one joke seems excessive, imo. Also, it would be hypocritical of me because I like Jimmy Carr and his jokes are way darker/more offensive, and have definitely included rape jokes at one point.
I think with Brand, making rape jokes is more offensive given his general misogynistic attitude to women. It's like the difference between Chris Rock making a joke about black men, and Bernard Manning making a joke about black men. Chris Rock you know it's definitely 100% just a joke with no malice or ill-intent - whereas with Bernard Manning you know that it's probably a "joke" but also his true feelings. Brand falls into the latter category - making "jokes" that actually are nothing more than a statement of what he actually thinks.
Having said all of that, I'm not sure about Alison Hammond anyway so that might be the decider for me.