I’m a bloke and I would agree 100% about there not being the same kind of physical threat. However rebutting a woman’s advances isn’t entirely without risk, they can set about annihilating you socially. I was also woken up of a morning in my student days with one girl’s very angry and aggrieved bf who had been spun a line about the nature of an interaction we’d had.
However I think I can empathise with the OP’s unease. Given it is still comparatively uncommon for women to make the first move, and when they do it’s usually because there have been significant non-verbal cues. We men can be a little bit dense about these depending on age/ relative experience. So I would imagine what’s running through the OP’s mind is simply what indicators may he have been giving off that gave the idea she could chance her arm, and if yes were they conscious?
I used to be famous for this sort of thing in my youth even when I was single, I’d have no idea, and when I was in a committed relationship even less so. One of my friends used to joke she couldn’t leave me alone for 5mins before she’d find me being chatted up by some girl or another. Thing is I just like people, and honestly I’ve never been exactly an oil painting so if a girl approaches me especially if she was a pretty one her being into me would be the last thing on my mind.
Sometimes that’s a dangerous blind spot as things can be escalating without even realising it, and it’s recipe for things getting awkward really quickly. I’ve never had my crotch grabbed but several times women have grabbed my arse, or just kissed me out of nowhere, and actually it’s really not nice if you aren’t “there” emotionally or physically with them, but we are supposed to just laugh it off.
I don’t think the OP is at all unreasonable to be feeling the way she is feeling at all, ok if it launches off into jealous/controlling behaviour that would be another thing, but personally if she was my wife and she had any frustration or question about my behaviour in an interaction like this I’d want to know about it and do my level best to put her mind at rest. NOT out of any sense of guilt necessarily, but because I would respect her feelings, and that has to include negative ones as well as positive ones. I wouldn’t necessarily interpret how she is feeling as an attack on my character/intentions. Simply that I’d told her an deeply unpleasant and awkward experience, and that she had questions about it.
Personally I’d want to have the conversation, and address anything that came up. Maybe there was an element where I’m somehow responsible, and I would want to correct any such behaviour pretty bloody quickly!