If I'm correct in thinking your abusive ex rolled out the whole "no one will ever love you, you're unlovable" line (how original, never heard that one before) ... Its possible you felt this relationship (is it the first serious one since the one with him ended?) had to succeed or it would prove him right.
But what he said is total bullshit so it doesn't matter; there are relationships that have legs and relationships that don't, due to a large number of factors ... it was a crap shoot which type you were going to encounter (in fact for all of us it's more likely we'd encounter one that didn't, simply because there are more incompatible people & situations than there are compatible), it means nothing.
As above, this guy is currently not a great candidate for a loving, committed, monogamous, settled relationship ... And in the long-term, even if he became one, you have the incompatibility around children. There are men who also already have children, don't want more .. and they are much more suited to you for a relationship.
You've got caught up in trying to make him commit to you, with "winning" to prove your ex wrong; but it's ridiculous because a. What your ex said is bullshit, and b. I dont think this guy would be a good, committed partner even to a woman who could have (more) kids with him, let alone when you're both aware that long-term he wouldn't have kids of his own with you.
Back on the loveability thing; I've been rejected in relationships/potential relationships (as most of us have) and I've rejected people.
If I think about ending the relationship with one guy, I would never ever have said something like "noone will.ever live you (because don't), yours unlovable" etc because a. I'm not a nasty, abusive ct and b. Because I couldn't possibly know that. Not about him. Not about anyone. Noone with a tap of sense would.
So your ex either doesnt have a tap of fkg sense and/or he was saying it to hurt and traumatise you, because he's an abuser.
Given we know he's an abuser .. that fits.
Just because someone (particularly someone with an agenda to hurt and abuse another person) says something, doesn't mean it's true. You don't need to prove him wrong, you don't need to prove he anything. He's clearly a damaged, nasty, fked up individual and what he says or thinks is flawed.