In our church you do whole shaking hands/kissing on the cheek thing. Nothing to do with physical attraction.
Any religion that says you can't touch or look at a member of the opposite sex in case they are unclean in some way, or likely to drive you to the heights of lust is puting a bit too emphasis on carnality. It usually puts one gender on the back foot with regards to what they do, where they go, what they wear, etc and no guesses which gender ends up with the short end of the stick, or the most rules and regulations.
I don't lust after random blokes or am driven to mad passion if a friend kisses me on the cheek or gives my arm a squeeze.
It cuts both ways - I won't put my hand an the arm of someone who - know would feel uneasy about it, if by the same tolken they won't squeal and rub themselves down with a brillo pad, assuming I am one step away from seducing them.
Why is it so hard for someone to just think 'bloody touchy/feely brits/christians/french' and shrug it off as not a deliberate come-on? Or even take a deep breath and explain why this is something that makes them feel awkward.
Some 'rules' are very difficult not get get insulted by - so having to pull a whole batch of paper cups from the bottom one only and not handle any others, then hand it to someone who selects one from the middle that you haven't possible have contaminated. Now that's a hard one to smile and nod to.
Of course there are people who just don't like being touched, for whatever reason and it isn't always obvious. Again, "please don't".
In the OPs case the touched man went on to shake another man's hand with no problems. I'm assuming the OP wasn't in Saudi or other country where the no touching rule generally applies? What if the OP was a black man and had the same reaction?