Did you witness any of this? Did you have any inkling? Your posts suggest you might have done as there is a lot of talk of him being “mostly” lovely. You need to realise that It is irrelevant if he is delightful, even 99% of the time if the other 1% he is physically and emotionally abusing your kids. Your duty as a parent, above all else is to care for and protect your kids from abuse. The only acceptable level of violence in any relationship is ZERO.
SS will not look kindly upon you leaving them in his sole care when you are aware of an accusation of abuse, regardless of the circumstances. You have to keep them away from him, for their protection and yours.
Please do not screw this up. You can’t go back. Yes it’s hard and yes he will make it difficult but you HAVE to stand by your kids. Going back is likely to mean you lose custody of your kids.
You need to realise how serious this is. Sending your kids to someone who you’ve been told has abused them makes you an accessory to the abuse.
Get some support. Women’s aid, social services, friends and family. There will also be women on here who have been in your shoes who can offer support. Don’t let him blackmail you. It’s just him extending the emotional abuse.
As someone who has seen families ripped apart in circumstances like this I would plead with you to take your kids and stay away from him. You have to keep them away from him whilst all this is going on. Keep talking to SS and do whatever they advise, which is likely to keep the kids away from him. You didn’t ruin his life, he did when he laid hands on your child.
You can do this but you need to stay strong and build up support around you. Keep going and one step at a time you will get there.