Happy - SS will not let your dc go to a violent father. No way. He may get supervised contract, in a contact centre, but that is it. He is in prison awaiting a charge of rape (and hopefully GBH against you) - your dc do not need this man in thier lives. SS exist to protect children from people like him! Really.
He may never have been horrible to them (YET! They are only small!) but they will have seen violence done to you. That is awful, and just as abusive and damaging to their small, developing minds, as violence against them. You KNOW this. If HE has influence in their lives, once your son is old enough to think for himself, he will start to copy his dad. Your daughter will assume it is right and proper for a man/partner to hit, rape and abuse her.
Break the cycle. You have already broken the cycle - keep them away from him, and any others like him. Surround them with people who lovethem, and want to develop and nurture their little, questing, absorbing minds.
You are doing so well. You really are. Soon you will be well, and living in a calm and happy household with your lovely dc. Aim at that - you can do it. You are doing it.
Keep saying that to your self. You can do it - you are doing it.