You don't seem to understand that 'hurting the children' isn't just your partner hitting them
Them seeing you hit each other is hurting them
Them living in an atmosphere of tension and fear is hurting them.
I would argue that a few years of living in a chaotic environment like that does Just as much, if not more, damage than if he did actually hit them. You don't seem to see this, only the impact on you.
You need to understand WHY social services are doing what they are doing. They are trying to protect your children, from your partner, but frankly, from you as well.
Take a long hard look at your relationship with this man and your relationship with your kids. Nothing in your posts has considered their right to grow up in a safe environment, only your feelings. Well, social services don't care about your feelings. They care about your kids. Your kids can't live independently yet, they have to be protected from him. And if you keep on going back to him, the kids need to be protected from you too.
We all argue, we all have our ups and downs. That's normal life. To get to the point the police are involved is not normal. You think it's normal and why is that? Because you grew up like that. History is repeating itself for your kids. Either you break the cycle or social services will.