It takes about 6 weeks for an initial hearing before a judge, so that will be end of January. At that hearing the judge sees if you can reach agreement, which you won't, and you offer (I imagine, guided by your solicitor) contact in a contact centre. It takes months more to get a full hearing at which he argues his case and you argue yours. Christmas is a week away so he can demand what he likes; you're in a refuge because of him and there is no way he can safely have contact with your baby. Don't worry about court, honestly. They are there to look out for your baby's interests and you have ample evidence of his abuse, harassment, intimidation and lack of knowledge of her routine, and your solicitor will make that case for you. You just have to give all the information.
I should add that in the absence of abuse, you're expected to ask for mediation in the first instance - there is even a box on the court documentation when applying for an order asking if you've tried it, and if the answer is no, most judges will send you back to try it before seeing them again. As he knows you don't have the money for solicitors and does not know you have legal aid arguably making court the first threat is a new form of intimidation. The fact he's demanding time over Christmas, a week away, certainly implies that as no court could create a contact order that soon unless it was an emergency enforcement (no existing contact order makes that impossible). He's just trying to scare you. The letter is just a letter, it's not actually any evidence he's even applied to a court, is it.
Courts DO NOT disrupt a child's care arrangements unless they have a really, really good reason, and the younger a child is the more that applies. They are not about to take a baby away from her mum to give to dad half the week when Mum is in a flipping refuge because of Dad - I mean, seriously. Think about it.
Far, she's a baby. You're in a women's refuge. There is a police report that you and your daughter had to be escorted past him while he sat in a car outside your home to intimidate you, a witness statement from your friend that he tried to break into the house, one from your mum saying he was slowly driving past hers after you asked him to leave you alone, and text messages abusing you before and after you said you wanted no further contact and showing he had no idea what her daily routine was. That's on top of your own statements of his prolonged abuse of you and his mother's and his abuse of your daughter. His solicitor will ask for whatever he tells her to - the judge doesn't have to pay attention to it. Please don't be scared by this, it was pretty predictable. The reality is that you can insist contact is in a contact centre until the full hearing, which won't be for months, and they can't actually force anything else without your consent until that stage.
Talk to your solicitor and see what they say, but stop feeling like he can dictate what will happen. I know he's made you feel that way for a long time, but it's bullshit. He can't force a judge to do anything and no judge is going to let him continue to harass and bully you, when you're a good mother and you are in a refuge and in receipt of legal aid because of his abuse and intimidation. Stop feeling like he can do what he likes and you need to placate and pander to him. All you need to do is think about what best serves your daughter's interests and keeps her safe. That simple, and so far, you're doing a great job - just as well, really. He isn't.