OP, please listen to your gut.
He is not whom he seems, but he is waving a big red glag that you would be wise to listen to.
You did absolutely NOTHING wrong.
You were straight up with him.
But he has decided to focus on this and go on and on at you.
Trying to make you feel you betrayed him somehow.
This is who he is.
An ugly petty insecure little man.
ICKY.
It will always be some or other stick to beat you with.
Work, friends, family, how you dress, behave, something, anything to cut you down.
He is the type of man who leaves women shadows/shells of their former selves without ever really knowing what they did wrong.
The regret for getting involved with him will be profound.
He cannot be who he is without cutting you down, one chip, chip, chip at a time.
Get the book "Why does he do that?" by Lundy Bancroft, it will explain red flags and put you ahead of the game.
Jealous, controlling, manipulative, coercive head fxxk is likely who he is.
I wouldn't give him the chance to apologise or try an hide who he is.
Just be glad he showed his ugly self so clearly, so soon.
Batshit, and unhinged.