For goodness sake - you have to do something NOW.
He is a time bomb, and the most worrying sickening thing you have said is that he is losing patience with your baby.
You NEED to do something before he shakes her or hurts her in some way.
DON'T say he wouldn't. He would. He might feel terrible afterwards and 'not know how he lost his temper' but it would be TOO LATE.
You are BOTH in danger. If today at the doctor's is your last chance to reach out before Christmas then PLEASE do so.
Tell the nurse what has been happening, PLEASE. Or, call someone - you say you have good support - please!! Call your mum. Anyone. Get help.
Longer term things can be worked out. You will NOT be depriving your daughter of a father. What you will be doing is possibly preventing a tragedy. He needs help too. You need to be apart. He is a danger to your baby. Even if he doesn't do something to her he regrets, you cannot stay in this situation - anger, threats of violence, being prevented from caring for your baby properly. This is all a disastrous situaiton for a tiny baby to be in and it WILL damage her.
Get out and start getting this solved. It can be, but not while you are both there being terrorised by him.