Okay, if you're still reading OP. It probably seems very unfair that he's come off the drugs and people are still being harsh.
Yes, it's a massive achievement, not many can even with this much at stake and he deserves lots of credit for that.
BUT it takes a long time to permanently change thoughts and behaviour to the point where no matter what happens in his life he doesn't take drugs. Not only that but most people who achieve abstinence relapse a few times first. It will take a long time of proving he can stay clean before he can make progress towards being trusted around the children unsupervised or able to reside with them (even without you there).
Then there is the separate issue that he has permanently crossed a line and changed everything by being violent towards you. He is capable of violence and always will be. The only way I would expect SS to be happy with him living with you after that is if Probation (assume he has a PO as you said it's been to court) after doing tonnes of work with him reassess him as being low risk to you. Why not ask them what they think of his risk?
It probably seems simple to you. He assaulted me on drugs, he stopped taking drugs so he's no longer a risk. Professionals will see it as he knows he is capable of violence after taking drugs, yet took them anywhere therefore every single time he chose to take drugs and go home to you he knowingly put you and the kids at risk.
As others have said this man has the capability of violence, not all men do, not all addicts do, this one does and it will take a fuck tonne of hard work and time with the support of lots of professionals for him to stand any chance of changing deeply in built patterns of thinking and behaviour permanently.
Meanwhile IF you risk it and live with him and either SS think he's not proved himself for long enough or he hurts you. That's it, kids gone, last chance for you to prove you can keep them safe and away from a violent environment blown. No notice, no more support just enough chances, kids gone.