I don't speak for all women, only myself.
I think that the perception that it's worse for women is partly because, typically (not accounting for very strong woman v smaller male), women are much less able to defend themselves. If I'm walking down the street at night and a man decides to attack me, I'm largely at his mercy - there is not the same discrepancy in strength between two typical men.
The statistic you're referring to is regarding random violent attacks, by strangers - it's true that men are more likely to be victims of those. Women however are more likely to be victims of sexual assault, stalking, harassment, and domestic abuse.
We also cannot discount the anecdotal evidence women have from their friends, sisters, mums etc, that won't be reflected in the statistics because they're never reported.
Of my group of female friends (including me), all of us have been subject to unwanted sexual contact in clubs/bars, some have been coerced into sex they didn't really want, some have been choked in the bedroom without consent, some have been followed when walking, some have been physically assaulted, and one has been sexually harassed by a male family member. None of these things have been reported, but they contribute to our perception of danger because the stories are everywhere. I don't know a single woman who has not been a victim on some level.
On the other hand from my male friends, I have heard stories only of 'fights' they were in. 'He's a dickhead', 'he came off worse' etc - nothing that left them in fear.
I'm not disputing that these horrific violent attacks happen to men - the stats are there. However it's not the same barrage of smaller 'incidents', which happen to women and leave them distrusting, and sometimes scarred.
This isn't even getting into the escalation of violence in porn, which women are well aware the vast majority of men consume.
People form opinions and fears based on their experience rather than stats, and stats can be misleading regardless.