The thing is, it's not about 'fault' or 'wrong'. Other than laws, there are no authorities to tell us what is right or wrong. I decide for me. You decide for you. He will decide for him. We will all be different.
You are responsible for your own happiness. Integral to that is you feeling that you can be wholly you, without criticism. You being yourself. You being free. So, part of your responsibility is to find people who have similar enough ideas about what's wrong or right, that you can be around them without feeling restricted in being yourself. Lots of people will think differently from you about lots of things. Lots of people might think that the way you do this thing or that thing is 'wrong', but that's according to their decisions, not yours. You will probably think that the way that they do it is 'wrong' too, and that's fine.
If you think that something someone does is 'wrong', that's a statement on your morals, not a statement on their behaviour, because you are not the authority on the matter. Nobody is. Nobody is in charge. They are in charge of themselves. You are in charge of yourself.
Same goes with your ex. He thinks you were abusive; that's about his morals, not about your behaviour. I could tell you that you were abusive to me because you ate a yoghurt last week. Would that make you abusive, or would it tell you something about how I draw my moral lines?
Spending time listening to others' negative views of yourself is not going to make you happy. It's not your responsibility to make yourself 'right' according to other people. It is your responsibility to choose your people wisely, so that you can live freely and express yourself honestly, without it rubbing anybody up the wrong way.
Un-choose him. He didn't take responsibility for himself; he spent time with someone he found abusive and he didn't leave, which would have been the self-respectful thing to do. It is your turn now to take the higher ground, and distance yourself from someone who criticised you and made you feel like a bad person.
Live by your own morals, not his, not anybody else's. Why would someone else's decision trump yours, with regard to who you should be, and how you get to behave? Do you ever get the authority to tell someone else what they are?