Right, so it's pretty clear that his behaviour is not about being a father, it IS about controlling you.
You can stop worrying about losing your baby, it's not going to happen. The fear is very real, I know that, but that's your hormones amplifying the fear rather than the fear being about a realistic danger.
Of particular concern - "I've always told him I would never stop him seeing his child but this never gets through to his brain he will just say 'can't wait until the day you push me out can see it happening' "
This is manipulation. By saying this, he is making it harder for you to say 'I've had enough, stay away from me'. And his manipulation is working - you don't want to prove him right.
I think you need to get proper legal advice. Hell, I'd be looking for a restraining order, personally. But you're not me, you're you. So sit and think, without considering what he wants - what do you think is best for your daughter? Make the decision, tell him what he's getting, and if he doesn't like it tell him he's free to take you to court. Go talk to a solicitor, or CAB, and take their advice.
On the details you've given, it's damned unlikely a court would give his residence, or indeed anything more that your already far-too-reasonable offer. And he doesn't actually want more, he just wants to hurt you and control you. Take the control away from him. Tell him to take you to court. I'd be very surprised if he did, because any solicitor he employed would point out to him that he's not going to get more than you have already offered, and he may well get less.
And start to think of the details of contact. As a newborn, if you're breastfeeding that restricts how long you can be absent from her. Supervised contact in a contact centre, no overnights, involving ZERO direct contact with you, always via an intermediary (the contact centre). If he can't harass you directly, he'll probably get bored and taper off.