You are right
He is wrong.
But that doesn't make it any easier, it's very unfair and extremely hurtful to be in this possition, essentially you can't win without coming out of it unscathed and harmed.
Your h is currently harming you.
This makes him a very unsafe person to know, personally I would be disengaging and limiting his relationship between you as strangers. It's like a game of chess with a million variables, you are drawn into a game you never expected or ever wanted, your peace of mind shot and your nerves so on edge you feel like your hanging over a cliff, the anxiety is colossal.
You just want the pain to end and the only one who could end it is the one who has stabbed you in the back and caused that pain.
You are clearly wanting mercy from his cruelty and he has up to this point denied that, in fact he's twisted the knife and blamed you for your insecurity. I honestly don't know the answer, extremely clever women have tried figuring the best way to stop these car crash infatuations and all roads seem to go nowhere.
They play out and only time will tell, if you seek knowledge or defend yourself he will call you crazy, it's par for the course, it really is a mind fuck and I hope in the future in whatever way you can, repay this favour back towards him in some way, shape or form, whether together or apart.
He deserves retribution for his cruel behaviour.
Consequenses should be now on your mind, you have already given him fair warning and he has ignored you, some men are like this, utter scum.
At least you know what he is now.