I feel guilty as he has had a hard time with his ex over me.
He has either done a right number on your head about this - OR, you have poor self-esteem. Or both: with a stronger sense of self, you would not have tolerated this awful man for 10 minutes.
NONE of that makes any of this your fault!
Why would you feel guilty over what his ex does?
You have done nothing wrong by dating HER ex.
Also bear in mind you only have his word for it.
And it's exactly the kind of bullshit story told by abusers - they ALL tell the same bullshit, the pattern is so marked that it is referred to as The Script.
I am suffering because of you, therefore you owe me
All my exes are crazy, therefore it's their fault I abandoned my kids
I am insecure, therefore you cannot wear that sexy dress when we go out
Your friends are Bad & Wrong, therefore it's my right to punish you when you want to see them
You make me behave better than usually, therefore it is your fault if I behave badly
Now look what you made me do
OP - can you please make an appointment with your GP, tell them about this man/this thread - & ask them to steer you toward appropriate counselling?
Again - not because you are in any way "defective". But because you have some worrying notions about what you 'ought' to feel guilty about, what you should be blamed for, & I suspect your self esteem - probably not great in the first place - (your previous abusive relationship would have seen to that) has been so brutally undermined that you have become far too easy to manipulate.
Please do the Freedom Programme, google/youtube resources about dealing with narcissistic individuals, & put all the energy you have poured into appeasing this frighteningly disordered man into healing yourself.
Dr Ramani has a wealth of hugely helpful videos - start here -
(HOW TO LEAVE A NARC RELATIONSHIP!)
Pay a lot of attention to what she is saying around the 6-minute mark. I think you will recognise your soon-to-be-ex here.
Get yourself booked onto this course - you can do it online -
www.freedomprogramme.co.uk/online.php
Buy yourself this absolute belter of a book -
www.amazon.co.uk/Woman-Your-Own-Right-Assertiveness/dp/0704334208
& learn, when dealing with dysfunctional personalities, how to Grey Rock, & how NOT to JADE -
www.medicalnewstoday.com/articles/grey-rock
outofthefog.website/what-not-to-do-1/2015/12/3/jade-dont-justify-argue-defend-explain
Also - spend some time (lots of time!) on the OUT OF THE FOG website where the JADE advice is. Because I am pretty convinced that your early family life contained some disordered individuals & you were not supported, validated, & encouraged to grow in confidence.
outofthefog.website/toolbox-intro
In short dear OP - so many of us have been where you are. Nobody is blaming you (or, if they are, they are dicks & you can ignore them), & while some of the PP responses, including mine, might be hard to read - please believe this is not scolding, it is TOUGH LOVE. Only you can ditch this man. Do it today.
Then start investing in yourself, beginning with the GP appointment, & the resources above. It's time to grow into the self-aware, confident woman you can be, who would not give jerks like That Awful Man the time of day.