I've been watching your posts OP.
You sound like I did. I was in an abusive relationship too.
My children no longer live with me because at the time, I minimised the risks of what was happening in our lives and I did make the mistake of not putting my children first. I lied and didn't engage with any of the help social services offered me. I told them we were apart when we weren't and numerous other things.
They WILL take your baby if they believe you are lying to them.
I see my children every two months now. For an hour and a half. Is that what you want? I'm still having supervised contact, although we're working towards unsupervised now.
Please for the love of god, don't make the same mistakes I did. Once it's done, it's so hard to change it back. You'll regret it for the rest of your life too. The guilt will never go away.
You need to cut all contact with this man. Allow social services to decide whether he should have contact. All the while you are facilitating him, you're putting your child at risk, and that is all that social services need to see.