I was about to say that I would first tell him firmly that he is NOT to do this, he doesn't get to tell you you're over-reacting, your body is NOT his property and your mind and feelings are NOT his to dictate. I was going to say that if he did it again, to slap his hand away hard and tell him loudly to stop touching you inappropriately, and if he didn't stop then LTB without hesitation.
However, having red subsequent posts it is very very clear that he is abusive. Sexual coercion is very very damaging (and I speak from experience). He has no respect for you as a person, he sees you as property to grope and fuck on when he wants with little or no regard to your personal space, wants and feelings.
And I know you'll be able to think of tons of examples of ways he is lovely and caring etc. which will make you doubt that he really is abusive. Abusers don't look like monsters, people are not 100% evil all the time. But he is abusive - don't doubt that for a second. Just because someone is only 5, 10, 20, 30 percent abusive doesn't mean that it isn't worth 'counting'. As time goes on you can bet your ass it'll get worse too. And I bet, if you start thinking objectively, he's abusive in other ways too.
And he (and probably you because he's conditioned you) will say he behaved his way only because he loves you, because he's a red blooded male, because he had a high sex drive, because he finds you so sexy and you should be flattered etcetc. None of these pathetic 'reasons' should EVER trump your rights over your own body.
It's very hard to get your head around isn't it? When you think it's just one little thing and suddenly the dam is burst. Take your time - you don't need to take any immediate action. Gather your thoughts, start viewing his behaviour objectively. The scales have started falling from your eyes and it's an emotional and shocking process. To be honest, it's only 2 yeas down the line after leaving a sexually abusive/coercive exH (who in many other ways was wonderful) that it has fully dawned on me the full extent of the effect of the behaviour on me. I hope you are ok 