OP, I am so glad you came on to MN for some help. (Although I really hope you're able to ignore the victim-blamers who are somehow finding fault with you for minding when your consent for various sexual things has not been given and he does it anyway, and then wants YOU to apologise for upsetting HIM. You really couldn't make it up.)
I hope you listen to us women who have seen it all before, who know how abuse goes and how it just gets worse and worse and worse the deeper into the relationship you go. And mark my words, nothing you do or say stops it.
You indicated above that it's not that easy to let him go, because you know each other's families etc. - and I get it. You also have a store of good memories and good times. You're still enmeshed to an extent, even though you don't have a mortgage or kids or are married (thank GOD).
So in one way, it's not an easy thing to leave the relationship....but in another way, it's the easiest thing in the world. Put your big-girl Nikes on and Just Do It.
Another word about abusive relationships: The entire reason they are so hard to leave is because of all the masses of sugar mixed in with the shit. The good times, the amazing memories, the times when he was so so lovely. After all, if someone was awful to you all the time, it would be an easy decision, wouldn't it? And this is what makes domestic abuse so appalling and destructive and so totally confusing for the victim. With someone who's just awful to you, it's upfront and you know where you are. But with domestic abuse, you are literally sleeping with the enemy. It's an extremely insidious, nasty, covert form of emotional, social, and/or physical aggression.
But you have to judge your relationship on its worst times. The times he has done things physically when you didn't want him to and when he wanted YOU to apologise for being upset. (My God, this makes me so mad, I literally want to slap him for you.) Those things which are upsetting you? THAT'S your relationship, your REAL relationship. That's the real him, not the nicey-nice one who lured you into this relationship under false pretences in the first place. That is, the pretence that a relationship with him would be a safe and nurturing place for you.
To put another way, a sugar sandwich with a tiny piece of shit in it's still a shit sandwich.
God, I hope you find the strength to dump this MF. I really, really do.
P.S. I am experienced with break-ups, and the absolute fastest way to get over someone is to have zero contact. No staying friends, no texting etc. I'm not saying getting over a significant relationship is ever fast, but zero contact is by far the best way. With zero contact, you'll find he crosses your mind ever less frequently, and when he does, your mind will kind of slip off him. But continued contact, of any kind, does nothing but prolong the agony. You can always say hi in the future if you want, but give it at least a year with nothing.