Not allowing ex partner to stay in my bed?

[FakeTurtle]

Might recognise me from past posts but I don't have many people to talk to.
I ended up leaving with our new baby to stay at my dad's because my ex was violent with me. I came back under the agreement he would stay in the spare room.
He wanted to have the baby overnight I more than agreed but he said he would only do it if he could stay in the bedroom not the spare one.
I don't feel like I should let him do that because we had terms that we agreed to. I brought this up to him and he started ranting (verbally abusive) until almost 4am in the morning. He eventually took baby and him to the spare room but baby was crying a lot and he started shouting saying stuff about me being a slag and a nasty bitch from the spare room. Then he stormed in the bedroom with baby and left him in here going on about how much he hates me.
He's made me feel like I have been completely unreasonable for not letting him be in the bedroom with baby even though the spare room option was completely available (apparently its not safe because the bed is so low down). I don't know what to do anymore, am I being unreasonable so I can put a stop to this unbearable situation?

He can’t guilt trip you for anything, you’ve done nothing wrong.
And no you can’t say he’d not hurt your child, he’s unstable.

Well done for getting out of there.
Do not go back.

Well done, op. Your baby has to come first. I've taught kids brought up in families with DV and they are harmed in a very real and lasting way by the experience. I would also refuse to let him have the baby overnight at all- it's not good for the baby to be separated from you for that long regularly at this age. Update social services on the situation and ask for them to arrange for your ex to see the baby in a contact centre so you don't have to be there and know he can't take the baby as a bargaining chip to get you back.