OP, you've been very defensive to anyone who suggests the way your relationship started has an impact on where you are right now. A couple of people were harsh, but most were not.
I truly mean well even though I have a hard time understanding why people make choices that hurt others.
Isn't it possible his guilt is affecting every aspect of your lives? He may feel that he (and you) don't deserve happiness. That you shouldn't get engaged/married because he didn't keep his promise to his fiancee. And guilt may also make him treat you poorly. Perhaps he didn't tell anyone about you because he was ashamed. He either didn't want his family to realize there was an overlap and your romance was the reason his engagement ended, or he was afraid of being judged.
Guilt doesn't just go away because you want it to. It may not mean he still loves her, but what kind of person would he be if he wasn't ashamed of hurting her? He may not be able to move forward while he feels this way.
This isn't a great analogy, but if I saw a dress I really wanted and I stole it from someone else, I wouldn't end up wearing it because someone might realize I had taken something that didn't belong to me. I'd feel too guilty to enjoy it. I'd keep it hidden in the back of my closet; can't bear to wear it but don't want to throw it away, either. After all, I really wanted that new dress.
You may not feel as guilty because you left your relationship sooner. But never discount that people are allowed to feel what they feel. Even if you don't agree.
Good luck. I'm sorry things don't look like they'll work out for you. And I truly mean that, which surprises me.