Ah so he is there when your son is there. Just a matter of time before he introduces his crackhead self to a frightened little six year old boy.
I really hope you booked the doctors appointment for today and that you're honest with them.
Stop worrying about whether social services will be cross with you and start focusing on the fact their priority is to help you to keep your son safe.
His welfare should be your priority if you love him like you say you do. So you need to cooperate fully with the police and social services to remove this man from your life as safely as possible.
The most dangerous time in an abusive relationship is when the victim tries to end the relationship. Which is why you cannot do it without police help.
This talk of phasing him out... it's madness. Absolutely madness. He won't go quietly and that's why it needs to be the authorities who remove him and put steps in place to keep you and your son safe.
If I was your sons ex and found out you'd moved this abusive drug addict into your home, the home my child visits, I would be calling social services and going straight to court to request you had no more access.
Far better for YOU to make the initial calls to SS and police so they know you're serious and genuinely want him gone and your son safe.