The past is the past OP. Don't torture yourself with thoughts of why you did and didn't do things. Don't waste your time trying to work out why he's like he is, or if he'll change - he won't. Draw a line, mentally.
This man is not your friend. He does not have your best interests at heart, nor that of your children. You will not be able to reason with him, there is no better nature for you to appeal to.
Your only priority is yourself and your children. Forget his family, at least for now. They have no legal right to see your children, and you have no obligation towards them. You said yourself your ex is nasty to his own mother, so they know what he's like, even if they perhaps don't admit it.
He has a history of drugs, drinking, being unfaithful. He's been physically abusive towards you, including whilst you were holding your baby. He's wished both you and your children dead. Didn't mean it, my arse - oh so that's ok then? He's clearly a nasty, dangerous man, with whom you need to break all contact.
You have a place to live, a supportive mum, two children, and a future to look forward to. Block his number, take yourself off FB and any other such social media. Tell his mum, briefly and factually, that you will not be in touch for a while as her son is harrassing you, the children are distressed, and you need to step away.
The final thing I would advise is do not enter in to any dialogue or communications with him. He's ignoring you - good. What good will come of any contact? He's an appalling husband and by the sound of it an equally bad father. He's already known to the police. If he comes around again, and makes threats/demands to see you/the children just phone them again. He'll threaten all sorts, but if you ignore him he'll soon get bored and move on to someone else. And no, he won't be able to take the children away.
Once he's out of your life you'll start to see things very differently.