That they're going to hear Ferrari say one thing they agree with but not recognise when he says a dozen things they don't - what does this say?
I suspect that's not what posters who object to this sort of thing really worry about, though. They may worry about giving listeners to someone they see as spreading bad messages, but mostly they worry that people will see a bunch of supposed feminists speaking positively about things this person has said, and then going off and claiming they are all a bunch of bigots or alt right or something.
Now, I think that is a mistaken strategy, but it may be true that people will make those claims and associations, look at all the other people who have somehow become part of the alt-right in recent years, it's crazy. Or people who have appeared at things hosted by pro-life advocates have been said to have some undisclosed religious agenda, which is wrong and bizarre on about three different levels.
So I understand why people feel uncomfortable with being associated that way, it seems like it could lose you sympathy and also just seems very uncomfortable for some if they are used to seeing themselves as really separate from those people.
It's a wrong approach though. It's wrong in how it treats the exchange of speech and ideas, which is profoundly important in a democratic society. It's wrong in nurturing, even unintentionally, this mindset that people who think differently are evil and stupid. And it's wrong strategically, because in the end you are like an insurance company that pays out every claim in order to avoid a court battle - in the end all but one voice will be deplatformed and there will be no one to hear you, no one who is pure enough, and your voice will be silenced as well.