Am I being unreasonable

[SillyCoralPombear]

so I have a 4 month old baby and me and her dad split when I was pregnant. We tried to keep things amicable for the babies sake, he came to all appointments, was at the birth etc.
since the baby has been born he has came to my house to spend time with her when he was off work for a few hours at a time, he is mostly left alone to get quality time and I only tend to go in the room if I need to get something or if the baby needs me. Things have began to sour between us and He has recently asked about taking the baby for his house for his time and while I actually would like this as it would be great to be able to have some me time she really struggles to settle with him. On most of his visits I end up having to intervene to settle her or assist with minor things with her care (such as telling him to adjust the babies head as the baby was flopped over etc) I don’t feel comfortable with her going with him alone until she will settle a bit better.
he thinks I am being unreasonable and is now threatening to go the legal route he does have a criminal record for violence in the past (not against a woman or child) and works shifts so can’t commit to set days to help keep a routine.
he also doesn’t show any interest in the responsibility side of things such as arranging/coming to appointments and says he was working so couldn’t come when I know 100% he wasn’t.
so basically I have 2 questions-
am I being unreasonable?
also has anyone been in a similar situation and gone through the courts and if so what was the outcome?

I think she's your baby and if you don't want to leave her then don't. I don't think I'd feel comfortable if I was having to give reminders about babys basic safety and care.
On the settling front though, my baby really struggled to settle with anyone else, he'd become absolutely hysterical and it was making me isolated and feeling like I didn't want to go out. It turns out that me being there was making it worse. If I pop out he's absolutely fine, he's only unsettled if I'm in the house but not with him. He's 5 months now and my parenta took him out for the day last week and he was perfect, so just something to bare in mind when you do feel you can trust dad to care for baby

She actually does settle for other people. She’s fine with family members of mine who might watch her for a short while I shower and things like that but on a recent visit it only took 20 minutes being with him before she was screaming. I try to leave her as long as possible to give him the chance to settle her but I usually end up having to intervene.

For the time being I would put my foot down. 4 months is still very young. Is your baby breastfed? No court in their right mind would order a breastfed baby away from their mum. At the moment all he has done is make a threat. How likely is it he will go down that route? It's stressful, expensive and time consuming. Even if he does you can be sure that nothing will happen overnight. These things take time, and in a worst case scenario your baby will be older by then. Stick to your guns. You can say no. You are already being very reasonable by letting him come and visit. The 'I can't, I'm working' excuse is a red flag. You don't get to choose when you are a parent. It is not the parent's right to see their child, it is the child's right to have a relationship with their parents. The conversation should always be, what is in the best interest of the child? If at 4 months he still needs his mum, then your ex needs to respect and work around that. Let him take you to court. You've already shown you are not preventing him from seeing the baby. Most likely scenario is he would get some time with him in the week (think a few hours at a time), increasing when he gets older. That said it would take months for it all to get sorted anyway. Stay strong, you sound like a great mum!