OP, I can sympathise a little with how you feel. When I met DH I categorically stated that I would never have a relationship with a man who had children, because either he would always put his children first, and I would get frustrated, OR he wouldn't, which would make him a rubbish dad and not for me. DH said nothing, but I found out a couple of months later, through his own slip-up, that he had a daughter. He lied again, said he had only just found out. I found evidence that he was lying, that he had lived with his daughter's mum for 2 years, and was still regularly visiting his daughter, a daughter he had kept a secret from me.
I was raging. Not only because of his deceit (like your OH, his excuse was that he knew I would dump him, and he thought he would enjoy it while it lasted)but because he thought he was cleverer than me- clever enough to deceive me- stupid delusional git!
I let him know in no uncertain terms that he was dumped. I was very angry and needed space. I also realised he had become important to me, but there was no way I could just accept it and carry on. I took a couple of weeks while I digested what it meant, what it would mean. Meanwhile, he called me several times a day.
I took him back, but I made it crystal clear that he was not smart enough to lie to me and have me never find out. He knows that I will ALWAYS catch him out, there is no point in trying to lie to me. I accepted that his dd would be part of our lives, with all that entailed.
Sadly that's not really what happened. THings went bad between him and his ex, unpleasantries were exchanged. The CSA got involved (which actually ended up with him paying less than he had been) and he discovered his dd was calling his ex's new partner "dad". We live a fair bit away, and DH was away a lot, sometimes for long periods, so regular contact was impossible, and things got worse when his mum (who had always been part of his dd's life) got a nasty letter telling her to stay away. DH decided to back off (a decision I didn't support, but which I believe he genuinely thought was "for the best") and we haven't seen his dd in years.
We now have 3 dc and have been married 10 years. Think it's a bit unfair to say that every man who has "abandoned" a child will do the same again. DH is older, wiser and in a relationship of his choosing, rather than one he felt shoe-horned into. We recently got in touch with his dd via FB, as CSA payments were stopping, and he felt sad. it is early days but they are e-mailing and hopefully things can improve from here (I had urged him to try to contact her for years, but he refused, although I know he always cared about her).
It's obviously harder for you, OP, as there are children involved, but I think the best thing I did was give myself a bit of space to sort out how I felt and what I wanted to do, what it all meant. I'm not exonerating your OH (smoking pot doesn't seem like a particularly mature way of handling things) but his treatment of these twins doesn't necessarily mean he will do exactly the same to you and your children, although of course it is a possibility. I think you should ask him to leave for a while until you sort out what YOU want, without him in the background. Good luck