Poor Dear OP - I know the kicking-yourself feeling all too well.
I know you've been careful to give them time, especially since they seemed annoyed, but it's been long enough. I hope s/he has just been busy, and isn't a dickhead. (And clicking a refund button is not much of an "inconvenience".)
When was the last time you contacted the person?
If it's been over two weeks, I would write them a brief note and tell them that since it's been so long, you've been advised to report it through Action Fraud, and that you have done so.
Mention that you can appreciate it's been a bother, but surely they understand you haven't done this deliberately. You were hoping to resolve it human to human, and you've given ample time for them to ensure that it was a legitimate error. (Avoid saying 'it's not a scam', and avoid using the word 'scam' altogether.)
Offer an apology once, and only once, for any inconvenience. And maybe drop an indirect reminder that it is illegal for them to keep the funds.
Also, perhaps you might try sounding somewhat more casual, not too terribly formal (for the time being, anyway) and write something more like, "I still can't believe I did that, really sorry.... I'll need to buy you a drink when this is over, haha..."
That sort of thing.
I've used this approach for similar things, and it lightens up the situation for the other person, as well.
Or perhaps he'll still be a dickhead.
Have you called your Visa card company, and spoken to an actual person? (rather than automated) If nothing else, they should be able to advise you of other avenues of recourse. You still might be able to declare it as an "unauthorized charge" even though you sent it. The person now is unauthorized to keep it. Visa might be able to reverse it. I'd call them a few times, just to be sure. Sometimes a different person who answers the call will be more helpful than others. (Ask me how I know.)
I'm sure you'll get it back. But the distressful waiting does suck a big one.
Meanwhile, if you don't contact the person again, they might think you're going to just forget about it. Let them know it's not dropped, and you've now filed.