I agree with Pattertwig and the advice to go to WA.
Don't try to salvage a relationship that is based on bullying and fear. In the long run, as well as the short run, your children will suffer just as much as you will.
Right now your H is threatening you because he realises he stands to lose his family life as he knows it. He knows what you're afraid of and he's using your cowed state (cowed after your relationship with him has had its predictable result) to bully you. This is standard proceedure for someone like him.
Uncertainty in this situation causes fear in and of itself for you -- if you've spent a few years trying to keep a lid on things through placating behaviour and have believed that you are in any sort of control of your life or of him (a common belief when a woman is living with a bully) then the threat of things being taken out of your hands and placed in the hands of the courts is a terrifying one. The courts are there for you just as much as they are there for him, though. And they are there for your children.
When (not if) you tackle the questions of visitation and access, please have a lawyer at your side. My advice when dealing with a man like yours is to leave no question or detail to be sorted out later when he is on the doorstep shouting at you, threatening bailiffs, etc.
I have a custody and visitation agreement that is over 10 pages long, probably closer to 20. Every possible holiday, birthday, and weekend scenario, including arrangements for substituting days, is covered. Times for pick up and drop off are listed. Phone contact with the children and how it is to be accomplished is all described.
There is also an agreed-upon person who is to act as mediator in case of disputes. There is a second tier mediator in case the first person's involvement proves insufficient (and to prevent exH from blowing his top and hauling me in front of a judge every time he feels miffed, to also prevent him from threatening me as a means of getting his own way in case of disputes -- my exH is a lawyer so this avenue would be no trouble for him, but trouble by the bucket load for me) In order to leave a controlling man with no way to control you, it's really important to leave no detail to chance in your agreement. Imagine every single scenario that might arise. Use a calendar when you're plotting your course.