I've been in a similar situation without the threats.
My advice to you would be, to cut all contact with him - would this be feasible? Do NOT continue a friendship with him or any sort of relationship AT ALL, because he will probably cause a nuisance to you and your child for a very long time, whether he changes his mind and decides to harass you for contact, then doesn't show up, whether he just argues all the time, tries to control you, or seeks to carry out his threats. This man is NOT a suitable father in any way, shape or form and you are best cutting him off like you would any other abusive bastard. Now.
Are you living near him, do you have mutual friends? This makes it harder. Please think about your options in terms of moving away if so, before you have the baby.
My actual advice is to either cut him off and never have any contact again or tell him you are going to terminate, then cut him off and have no more contact ever again. And obviously don't terminate because you do not want to, or have to, and you have done absolutely nothing wrong.
I am sure others will flame me for this advice but it is now about your happiness and security and safety, and that of your baby, and nothing to do with his rights at all as he has negated them entirely with his appalling behaviour.
I am sorry you are in this position but the best way is to get him out of your lives permanently ASAP and never look back. 