I do wonder if guilt can be caused by the fact that the sexual component of the assault confuses the way our minds process an assault, in the way that a mugging does not.
While I couldn't possibly speak for anyone else who's ever been a victim, this is most certainly the case for me. There's always this question of did he just find me attractive (nothing wrong with that per se - it happens, even at work) and express it in a clumsy way? Did I lead him on because I was bubbly and personable as you would want to be as a junior analyst still on probabtion? Is the difference between a fling with your boss and harassment whether or not you find your boss attractive?
But then it isn't, is it? I'm a boss myself these days and have been for years. I've had male employees as well as colleagues and bosses who've obviously fancied me, and some I may have fancied a little myself and might have entertained getting involved with had we met under different circumstances.
But we didn't, so I've always kept it professional. It's a fine line to walk at times between meeting up with a colleague at the hotel bar for a drink and a good laugh while on a business trip because you're just both really bored and flirtation. But it's a line, it's there and I can honestly say I've never crossed it as a boss, and none of my other bosses have ever crossed it with me.
So, yeah, I think you're right about the sexual component making people feel a different kind of guilt. Also, I think they have no rational reason to.
FWIW, I've had a very decisive word with my bosses on the matter of ex-boss, and it basically went like "he's not coming because I say so. I, not you or HR or anyone else. I have the authority to pick my own team and I will make this decision by myself. I'm sorry if this makes you feel uncomfortable, but I can't have you treat me as an HR case in this matter because I've already been victimised once, and I need to not be feeling disempowered right now".
Only response I got was "perfectly fine, next time don't include the word 'sorry'". 
Obviously, ex-boss' manager was a tad less delighted about my hard no, but at the end of the day it's my initiative, so he doesn't get to staff it.