so it's not like he hasn't done these acts to me before so why does it bother me
Because you have boundaries, whether you choose to listen to them or not.
How old were you when you were first touched without wanting to be, OP? Totally understand if you don't want to go down that path on a thread, but somewhere along the line, you've been taught that when you feel bad, you need to shut up about it. And that's why you're confused: your feelings are telling you one thing, and your brain is telling you to cover it up at all costs. Your brain/your conditioning, are trying to overwhelm your heart.
This won't have happened due to your adult relationships. You will have learned this young, and your adult relationships will have followed the pattern you were then familiar with: Feelings say 'no', but I have to say 'yes'.
You don't have to follow this pattern any more. Feelings say no? SAY NO. And remove anybody who doesn't respect that 'no' from your life, altogether. No ifs, no buts. There is no excuse for overriding your feelings. Not for you, not for anybody else. Currently, you over ride your own feelings by allowing this to happen. What will he do if you say no, and stop him? Are you afraid of him, physically? Would you be in danger?