Yes, I've acknowledged that in a previous comment - while the specific example was fair, the unfounded generalisations weren't.
Personally, I'd also be saddened by someone calling my parents 'pervy' if they had dementia, and sexually violated people. But I wouldn't be upset with a member of the public, visiting their family member, slapping my parent's hand away if they groped them. I'd be angry with the care home for allowing that to happen, to both the victim and my parent.
I do think that OP should push this issue with the care home, and escalate the issue by reporting them to whoever the appropriate authorities are, if need be. It does seem counterproductive to let it go, and not do anything more, meaning there may be more incidents to both her, or others.
Posters on this thread seem to be forgetting that a member of staff witnessed both the groping, and OP's instinctive reaction, and their response was to scold her and not take any other action to ensure no repeats of the behaviour. So clearly, it's not a well run facility, if they're just brushing off assault.
If they're shrugging off male residents doing that to visitors, what might the male residents be doing to vulnerable female residents, who can't advocate for themselves? So for that reason, I hope OP does complain higher up. Perhaps she worries that there might be resentment that's taken out on her grandfather if she does so? But she still should, really.
But either way, I don't think that slapping away the hand of a grown man (who is stronger and more physically intimidating than a tiny child, and whose violation might trigger a trauma response) is unreasonable in the moment. I have sympathy for OP's reaction, and I might react the same way. A violation isn't any less triggering or distressing just because the perpetrator is unaware that they're doing anything wrong.
I think it's possible to feel OP was reasonable - but should report it - and that the man is not technically at fault, that neither of them are being safeguarded, and that the care home is the entity at fault here.
I also think that if these patients are wandering freely instead of being in a dementia ward (what I've always experienced in rest homes where I live), or if a visitor is accessing the dementia ward, then they should be warned ahead of time that several of the residents may attempt to be inappropriate, and either escort them to their relative's room unmolested, or make it very clear how to deal with any approaches. Forewarned is forearmed.