Tell him to "Fuck OFF"
You are not going and the kids are not going. That is it!
Stay home, have a UK holiday with your DC, and have fun. Call the other couple and tell them exactly why you are not going, tell them he is abusive and has bullied you for years and that now you have re-claimed your life.
I took my DS to North wales last year for a week, it rained pretty much every day, it was FREEZING, but we had none of the tension, we got to spend really great one-to-one time together and it was just the best time EVER!
Put yourself and your DC first, why does an abusive bully get to tip you out of the equation, but get to gloat in the glory of taking the DC on a holiday abroad?
Remove the rug from under his feet. Whatever he wants you to do, do the opposite. RESIST any and every attempt for him to set the agenda. Upset the apple cart as often as you can. Make him squirm, embarrass the shit out of him and tell EVERYONE about his abuse of you.
Gloves OFF now love, time to go to war! 