Agree with Lonecat. Don't put up with this for your own sake but also for your children's.
Women's Aid can give you advice and hand holding while you go through the process of drawing up a formal contact agreement. Also they may be able to recommend solicitor who has experience of this type of behaviour. Be aware that mediation with someone who is verbally or physically abusive can be difficult as they may use this as an opportunity to rant and rave at you. If so ensure the mediator puts them in their place - you should all agree before hand the rules of the session, no shouting, focus on the children not the past relationship etc.
A good parent would do anything to see their dc, so if he drops contact just because you've annoyed him by trying to set up some structure, he's not a good parent. Unfortunately no-one can make their ex a good parent, you can only try and set boundaries and be the best parent you can be yourself.
From my own personal experience of mediation/solicitors you would be supported in trying to organise a formal frame work for contact so that your DCs know where they are going to be when. This is going to ruffle feathers, but is worth pursuing as it will make things better for your DC and you. Your Ex is currently exerting unreasonable levels of control over your life preventing you from moving forward and making a new life. I would suggest seeing a solicitor first they are probably likely to suggest mediation. It is worth at least trying attempt mediation so you are shown to be trying resolve this without involving the courts. I found the mediators to be very pleasant and to listen to everything I said and understand that children need regularity and structure in their lives. I know tackling it is really hard, but from the sounds of things it will be worth it for you and your children. Your EX currently wants everything his way and this can not continue.
I have 2 DCs and for the past 2.5 yrs their delightful father has pretty well detached himself from their lives.
For 18 months after I found out about the affair he lived in the spare room, then last year he moved out and straight in with OW and her 2 DCs.
Contact has been confined to 14 overnights and him picking eldest DC from school and then running back to new home 2 hrs later. Contact is random, I am not allowed to know anything advance, eg, he phones at 1000 in the morning and says he will pick them up. Thursday PM he phoned to say he wanted then on Saturday night would pick them up about 1700 on sat and could I collect on Sunday about 1600 - 60 miles away.
I am so tired of him treating the DCs so badly and feel like a doormat,I try ensure that they maintain contact, they adore him but the lack of planning and hence total control he ahs over my life, he knows what I do, when, where how etc is driving me mad.
He is very verbally abusive and threatens me with all hell and damnation if I ruffle his or her feathers.
Eldest DC 6, spoke to me this evening as they are with Dad and told me he did not want to live this life anymore, he did not want to stay with Dad and OW- my heart cracked. how sad is that?
How do I get control back without hurting the DCs.