Sisiwawa
I was in a relationship once with a man who was 'troubled'. His controlling and manipulative behaviour was quite easily explained away by a couple of factors. It didn't excuse it but it did explain it.
He didn't mean to be abusive but his behaviour was. The impact on me was no different to it would have been if he'd done all of it intentionally and was an utter arsehole. It took a friend to point this out to me.
That's where the mindshift comes. It doesn't matter how easily explained his behaviour was, the impact on me was negative. It made me anxious and placed restrictions on my movements and my behaviours. It controlled me.
Point is, it doesn't matter if its intentionally abusive, learned behaviour, a feeling of inadequacy, anxiety, insecurity, defensiveness or whatever. It doesn't matter what the reasons or the explanations.
If you experience it as 'abusive' then it is.
In your case, your partner is telling you that your relationship is not worth his effort. That you happiness, your peace of mind, your security whatever are not worth a bit of effort or discomfort on his part.