The argument inside you is coming from your heart (the true you, the one who has always been silenced since you were a kid), and your brain (the you that's been conditioned to do what you're told, regardless of how you feel about it)
Sometimes your brain wins, and you go to this man for affection and support, because that's the way you've been taught a relationship works, and this is the relationship you're in.
Sometimes your heart wins, and you want to scream 'GET OFF ME, DON'T TOUCH ME, I DON'T WANT YOU NEAR ME', because that's how you truly feel, when you're not trying to 'get it right'.
Life isn't about getting it right. Life is about doing things that make your heart happy. The flaw in your thinking is that there's something wrong with you. If you stop thinking that, and start asking why you would feel all the 'GET OFF ME' feelings if it's not to do with a fault you have, you'll start coming to the truth: Your instincts are screaming at you to keep your distance from this man, because he has overstepped and disrespected your boundaries time and again, and the reason he's been able to do this is because you've let him.
how can i get over all this and sort my head out
Put as much distance as you can between him and you. It's that simple. There's nothing wrong with you. Don't ever try to change yourself. Change the people you keep company with, so you're surrounded with people you feel good with. That's all you ever need to know about boundaries, and self respect within relationships.