I don't think that fantasy is about the violence, or power specifically though. It's about safety.
I've been sucker punched in the face twice in my life. Once when in University, and again a couple of years ago. In both cases it was just the one punch, and I broke my nose. In every other way the two experiences were very different for me though
The first time I was in no real position to defend myself, and it really shit me up for months afterward. I'd go over it again and again in the middle of the night. Despite the fact that I'd managed to get my attacker to run off (I'd somehow come back up off the floor with a half brick in my hand and lobbed it at him before my brain had even registered what was going on) I still spent nights replaying it, wondering what I could have done differently, fantasising about beating him up, getting him arrested. Even killing him sometimes. It wasn't about revenge though, or wanting to feel powerful. It was about not wanting to feel scared, of not wanting to get hurt again.
The second time it happened I was never in any danger. I'd gone from being a skinny beanpole to a bit of a unit. (Don't get me wrong, I'm not a fit man, think Greg Davies for reference. I'd be useless in a proper fight but I could probably overpower and then sit on most people these days.) Plus the guy who'd hit me was probably about 70. Yes he hurt me, but I wasn't in any real danger, and it didn't scare me. I was a little shook up for a couple of days and my nose was sore for a while, but I didn't relive it over and over again. There were no fantasies of beating him up.
You're right that even the most non-violent men wanting to be able to handle themselves, but it's not a "I want to be able to beat someone up" thing, it's a "I want to be sure that I won't get beaten up" I don't see walking away as the less desired option, or a "loss of dignity". Its the preferred outcome for me.
For what it's worth. I'd much rather be a bloke than a woman. Yes, there's a chance of random violence from strangers for no good reason. But I'd much rather that than have to deal with the shit women have to put up with on a daily basis. I can walk down a dark street at night. I can go round a potential partners house and not have to worry if I'll ever leave again. I don't have to worry about the dichotomy of the sex I'm attracted to also being the people most likely to do violence to me.
Violence isn't something I have to think about day to day. In fact, I can go weeks without it crossing my mind (ignoring all the beating people and shooting people going on on in TV show / films etc). Yes, every so often it intrudes into my life out of nowhere, but I don't have to worry about it day to day, not the way women do.