I've only read the OP's posts, so will undoubtedly be repeating other people here.
"... he looked shocked initially ... Then he randomly muttered that I’m clumsy etc and I said it wouldn’t have happened if it wasn’t due to his actions ... He got defensive and said he wouldn’t apologise ..."
His reaction here is one huge red flag. For me, the reflex reaction would have been 'Oh I'm so sorry are you OK??' - let's face it, for most people the apology would have been out of their mouths before they'd even considered whether it'd been their fault or not. But no, this is not his reflex, instinctive action. His reflex is to blame you and make it clear he is NOT sorry. That's the red flag right there - takes no responsibility for his own actions, deflects, blames others, digs in stubbornly.
"I didn’t want to and was trying to get away from it since the minute it started. I can’t stand that kind of stuff."
I was going to ask if he knew you can't stand it, but really it doesn't matter whether he knew or not. "He got defensive and said he wouldn’t apologise". He's wrong already for making it your fault so it doesn't matter that he'd be doubly wrong had he known you dislike that behaviour.
And it's entirely reasonable that you "can’t stand that kind of stuff." Had you not liked - oh, I don't know, being given a present at Christmas, THEN he might have been able to claim that your dislike was unexpected and threw him; but not that many adults like 'play fighting'. I'd hazard a guess that it's a small minority. So your not wholeheartedly participating can't really have been a surprise, can it? So he can't claim his response was the result of being confused, he really can't.
No, his response smacks more of being pissed off that he can't do what he pleases. Pissed off that you told him no and meant it. Pissed off that you have a say in what activity you participate in. Plain old pissed off, and blaming you because he is the centre of the Universe and how dare you thwart his desire to do "something stupid like put his finger up [your] nostril or something". He's childish and self-centred. And not willing to take responsibility for his own actions. Actually, let's come back to the not taking responsibility for his own actions, because this is central.
If he won't take responsibility for his own actions - who is to be responsible for the consequences of his actions? Someone has to be - and he has nominated you. He can do whatever pleases him (and him only) and you bear the responsibility for it. You can get hurt by his actions, and it's your responsibility that you were hurt. Your fault. Are you seeing where this is going yet? Are you seeing the disrespect? You getting hurt matters less to him than him having 'fun' (otherwise, he'd have apologised the second you left the sofa).
Now it might be that he's just an arse. That's actually the Best Case Scenario. There are other scenarios.
Worst Case Scenario is that he's an abuser at the start of the abuse cycle. He's done the Nice part of the cycle, presented as a regular bloke, got you liking him. Now it's time to start the slow slide into the Nasty part of the cycle, which begins with testing how fast he can take it. What will she accept? Can I upset her, and get HER to apologise to me for having boundaries? Because for an abuser, that's the aim. And having got that, he's confident he can push it further. He's almost there - you've told him "I know he didn’t do it with the intention to cause harm" so you've absolved him of deliberately hurting you, a half-apology from his point of view. And how does this statement gel with Worst Case Scenario -
"He seems very defensive and not apologetic at all. Almost as if it makes him appear like a woman beater or something (which was definitely not what I was implying)."
No, you weren't implying it; but I think it is time to consider it. Because although it is Worst Case Scenario, it is a shockingly common scenario.
And remember - Best Case Scenario is that he's just an arse.