What Cat said with bells on. I was once like you OP, eternally optimistic about people, gave so many second chances because they appeared remorseful (only when I had spelled it out to them that what they had done was unacceptable and they had to pretend to understand this to keep me in the relationship).
I have had many bad relationships with narcissists. This hasnt made me bitter, this has made me learn the pattern and I'm now so grateful for these lessons as never again will I tolerate the behaviour you are describing. I can almost guarantee this guy is a narcissist just from what you have said. You may not be ready to accept this and I fear you will find out the hard way like many of us on here.
The fact that his past relationships ended badly and that he is now excusing his behaviour because of this is a HUGE red flag. Narcissists always say their exes were crazy, mean, cheated on them, were unreasonable, took their kids away and they will not accept their responsibility in any of this (most healthy people can reflect on the part they played in any relationship). This serves 2 purposes. 1. he gains your sympathy and you begin excusing his over the top creepy behaviour because he has been hurt in the past. 2. You will never seek out the truth from his past and always see him as the victim and even come to defend him when you hear the truth (you become his flying monkey).
When you say he broke down, do you mean he cried? Classic narc tactic to get you to feel sorry for him and stop asking questions. You have basically told him exactly what to say to you to keep you where he wants you.
Most narcs do have good points (in the beginning). Its how they get you hooked and you spend the rest of the relationship chasing this nice bit which is gradually replaced by gas lighting (hes already started this on you), put downs, silent treatment, isolating you.
For those who haven't been in these relationships, it may seem like he has apologised and is sincerely sorry. For those who have, we know how this story plays out and are warning you because it is so so obvious to us just from the little you have written.
If you continue to see him OP, please keep him away from your DC and dont tell him anything too personal that he can use against you at a later date.
No matter what I can tell you about how he is displaying behaviour typical of abusive men, the biggest advice I can give is trust that niggling feeling in the pit of your stomach which is telling you this isn't right. This is your gut instinct and it is never wrong. Pretty much all of us who have been in these relationships have had this feeling and ignored it at our peril.
Life will keep teaching you the same lesson over and over until you learn from it.