Name calling is a sign of weakness if you're trying to argue your case, but many TRAs have no interest in arguing a case, only in silencing their opponents.
You're right, and of course there are a lot of feeble-minded people who will decide, "Person X says this group of people are fascists, therefore they must be fascists" and then they don't apply any more thought to the matter because thinking hurts.
Still, I'd like to believe that people who have more than an ounce of sense - and surely there are some out there - will see the word "fascist" being bandied around and see it as a sign of desperation from a group of people who have nothing but insults.