Charl, the line that resonates in your OP is "...I'm happier than I have been in years". You must have worked hard to get to this point given all he put you through? It takes some people years, post affair, to feel so positive. If you are happy, your son will be too.
I have experienced similar, I also had the opportunity to have X back. Still do as it goes! I'm alone with four children, two under two. I understand the work involved in progressing towards independence and self confidence and the second, then third thoughts regarding 'what to do for the best'.
FWIW, your X sounds confused and I'd agree, emotionally unstable. My concern for you is that by entering into something with him, at this stage, his flakey, negative behaviour might drag you back down too.
Again, I do believe you've worked hard to arrive where you are now and it's almost as if he's looking to hitch a lift on the back of your strength and ability to recover, to snatch a breath of the happiness you've created for yourself and your DS, rather than approaching you and a potential rekindling of anything with you, with his own offerings of positivity, stability and clarity.
This on top of the negativities you listed in your OP?
Just one perspective but I can't help but think that if with him is where you're destined to be, you would not be struggling so hard to see it.