He doesn't see you and your dc as his "family" as he still prioritizes his family of origin. There is not much emotional maturity or emotional intelligence there at all. Nothing came of the casual engagement because that would make things a little too real for him...yet he has no trouble doing the deep discount regarding the existence of two children. 
His family have put you down and he sides with them. He is a blind weakling.
The "f+ck you" gesture is childish. That he hangs into it and treasures it speaks volumes about his choice of focus on his self serving entertainment. Shallow shallow bloke (Can't call him a man).
When he has two small children, yet he still volunteers to be in the presence of drug users (whether or not he is using himself) is just stupid. Stupid!
Let him go. The emotional connection is already broken, if it ever existed in the first place. The financial aspect can be stressful, but with the advice already given- you will be ok. It won't take long to absolutely thrive. You may believe that your financial security lies with him...But it does not. Presently just from not being married to him leaves you in financial risk. But from the parade of Monumentally Stupid Decisions already produced by this bloke, who knows what bat shit crazy circumstances would be in store for your future?
He is not a keeper. I agree that the argument/breakup is to green light sex for his gratification. Therefore, please, no make up sex on his return- no matter what he says, it will just be lip service with a validation stamp to play that script again and again.
It would serve him right to have his stuff moved to his mummy's for his return. I am with AF, it is trip=break up.
Sorry you are going through this. And I'm
on your behalf that he walked out on you shortly after giving birth to dc2. He is truly a bloke of no character.