I had this. did it to me once, i told him I didn't like it, that it upset me and made me feel bad.
He did it again. I told him enough, and ended it.
He didn't listen then either, took 2 visits from the police to get him to stop texting.
This man is showing you that he will use any manipulative trick he can to get you to behave the way he wants you to, regardless of how it makes you feel.
He is a dud, and potentially an extremely abusive one. Don't do what you want to do here, do what you MUST do, which is end it. For your own safety, security and well being, there is no other option than to turf this guy out of your life forever.
Ignore this advice at your peril. It could result in decades of lost life, lost friends, you could be assaulted or worse. It takes on average 2 years for the abusers nice act to wear off, he can't manage to be nice for more than a few months.
I too have a drawer full of t-shirts and look back on a life where I realise that I have never known a proper loving relationship. I have a son who has an abuser as a father. I regret this. he deserves better. I deserve(d) better too.