Yes, you can't find someone guilty of rape, solely based on the fact that they were previously accused of rape by someone else can you?
"Similar fact evidence". Not solely the accusation, no. But if several women say a bloke chatted them up in a pub, offered to walk them home, then brutally raped them when they reached the door, then the chances they all invented those facts independently, and were able to tell the police a course of very similar, coherent, and systematic events, look slim. And knowing of a previous conviction or accusation can make a victim feel either emboldened or obligated to report her own - look at what happened over Yew Tree.
Of course it's horrendous for men falsely accused. But it's pretty horrendous for women truthfully accusing, and failing to get convictions - especially when the bloke then rapes someone else, too. There is no right answer with this one - it's a question of which victim you want to prioritise: innocent men accused of a horrific crime, or raped people (boys and men are sometimes victims, after all) denied the supporting fact evidence of other victims, and thus justice.