I find it extremely hard to digest men are bigger/stronger therefore women are smaller/weaker comments. build, strength or gender does not give carte blanche for unreasonable behaviour!
I found it hard not to comment here as, within my job, i carry out fact funding investigations into HR related issues.
I hope you take these comments in the spirit that they are intended OP, in that i hope they are helpful and provide some insight.
Firstly, you feel hard done by, no one can argue with your feelings so take them as granted. But please be objective enough to know that your emotions have played a part in the language you use when writing your post.
you have used a lot of emotive words which, naturally based on how you are feeling, assume you as the victim and your colleague as the aggressor.
I would urge you to report the "incident". as you cannot report "him" without telling the story!
Please take a few moments and perhaps write a paragraph or two noting only facts. take out moral stance, emotions, background issues. so it would go along the lines of (based on your post)...
colleague approached me with a work related matter. i did not appreciate his tone. i stated "who do you think you are talking to" and made contact with his person using a newspaper i was holding. colleague then made contact with my person with his open hand. end of interaction.
force is relative to the individual. in this instance, if you can be objective as possible here, we aren't talking about you extending a friendly, warm handshake/embrace and being thumped in return. equally, would you have been as perturbed if you'd made contact with the newspaper and being upset if he'd hugged you in return? each interaction is a clear invasion of your personal space EXACTLY the same as the newspaper making contact with him. hence why force/intention is clearly relative to each individual what they find acceptable.
As a side note, i have a colleague who has a
diagnosis on the autistic spectrum. he informs everyone who is new to the team that he does not understand social ques as sarcasm AKA banter is simply outwit his understanding/recognition. he is a very literal thinker, "take a seat and i'll be with you in a second" literally means 1 second to him. is there any possibility that your colleague cannot/did not read the social ques of your "banter"?
I also have a colleague who trains in a boxing gym. her reactions are therefore lighting speed, i once accidentally gave her a fright from behind a doorway, without a milliseconds thought, she lashed out and slapped my arm due to fright. it was bloody sore, neither of our faults, wrong place wrong time!
jeez, sorry for the long post 
i hope you are able to resolve the matter one way or another.