@fruitbrewhaha
Of course it's inappropriate.
He knew there was no hole. It was a clumsy way to talk to you. And even if there was a hole, he didn't need to touch you. He could have pointed to where it was without touching you.
You wouldn't tell a man he had a hole in his shirt and poke your finger in where it was gaping in between the buttons.
He is a creep. I'd bring it up with him.
I am with this. ^ It was an excuse to touch you. What a fucking creep! He knew damn well it wasn't a bloody hole in your dress. What a sleazebag.
Bet he wouldn't have done it to a man. 
I don't think there is much you can do about it now, but next time, poke him in the nose, and say 'touch me again, and I'll have your knackers on a platter you fucking creep!'
Men like this set my teeth on edge. The women on here who think it's OK that he touched her skin in this manner, have clearly never been groped or catcalled or leched at or sexually harassed. Lucky you! 
Anyone who thinks he was just being 'friendly..' are very naive. As has been said, he did NOT need to touch her.
@halwitpicker
Don't be polite next time OP, and ignore the posters saying 'total non-issue' and 'can't men do anything now without it being seen as an attack?' and 'I couldn't get worked up over this.' Bully for them! Doesn't mean others should not be pissed off by it. And do bore off with the 'what a hysterical reaction' comments! What a misogynistic piece of shit that is to say! Christ on a bike! 
Women don't need to be nice and compliant, and put up with shit that makes them uncomfortable anymore. It's not the fucking dark ages! Ignore the posters telling you YABU and you need to get a grip. They have no right to tell you how to feel.
And as for comparing someone reading over your shoulder, to a man poking his finger through your dress, and touching your skin without your permission. Words seriously fail me
!
Some people have very low standards.