I tried to explain I’m not saying he is a bad person.
But lots of the time he is emotionally and mentally awful to me
The real problem is that he is a bad person, or at least a very damaged one. I'm sure he has a whole list of reasons 'why he is how he is' but at the end of the day, he knows he's a shit partner, you know it, and he knows you know it. That's what he's angry at you... because you reacted in a normal way to his abnormal behaviour, and he feels bad because it reminds him of his issues. But he's too vain or selfish to say 'Hi, I'm the problem, it's me', so instead he makes out that you're the problem.
Leave this fucker. He doesn't deserve you. Be honest, the only thing really keeping you in the relationship is the fear of how badly he'll behave when you decide to leave. He almost certainly also knows that's the only reason you're staying too (not that he'll ever admit it), but he's too dysfunctional to change his behaviour and at some level he understands that as long as you are scared of him, it stops him from being abandoned.
He's not scared of being abandoned because he loves you, he's scared of being abandoned because it confirms that his behaviour makes him unlovable.
TL;DR Him acting aggressively keeps you under his control and in the relationship, which is what he wants, which is why he will never stop acting aggressively.