What do your dc benefit from being left with this man. If the answer is nothing or he is a danger to them which from what you have said is the case then you can't possibly just hand over contact. Without a court order in place without wanting to frighten you there is also no firm boundaries of returning them on time etc. Also without a court order supervised visits are very hard to get and you will likely have to pay for them yourself. Let him take to you to court, I suspect it's all bluster, it's another way to frighten and control you.
In the mean time collect all your evidence, you do have evidence you have police reports etc and though no charges have been made everything is logged.
I'm Assuming ss contact you after police are called as they have a duty to when dc are involved. So they should have everything logged to.
I strongly advise you ring domestic violence, they will find you a refuge. Although not nice to begin with you will be safe and as long as you don't tell people where you are he won't find you. They will try and rehome you in the new area asap.
I know it's a very scary time for you and the easy answer is to just give him what he wants but you know that's the worst possible thing for you and dc. You are their mum and you need to protect them the best way you can. You will get lots of great advice on here, please take it.
The most important things you do now is to log everything, report everything and either video or voice record what you can. You also need to recall domestic violence and get yourself moved asap. I know that's distressing and you are leaving behind a life and friends and family but your inner circle can contact you, you and dc just need to be safe and away from this man and family. That's gives you more time to evidence everything, write a timeline, you can rebuild, you are strong, you ltb so you can do this.
Please get the help you and dc need to escape this man, yes it will be hard at first but I assure you life will get better and you know when your dc are older they will understand and they will see what you did for them. As adults they will not thank you for the future this man will give them. They seriously won't. But they will be proud of their strong mummy for getting them out when she did. Sending you
and strength. You have already done the hard part of leaving him, see the rest through and give yourself and your dc the safety and stability you all need.