This really is the entire crux of it. No-one called that man out, because they were scared of the confrontation.
If it was a transwoman, then yes, you might get accused of a hate crime. But that is going to be the second, third, or last thing you think about when standing there. The first thing you think about is are they a threat? Will they become a threat if I say something?
Anyone who has been subjected to street harassment knows full well how it can escalate if you retaliate.