I understand your frustration. You've had the 'work' and he's had jack-shit as far as responsibility and costs. But as frustrating and infuriating as that is, you simply cannot put your children at risk (even a minimal risk) for the sake of forcing him to 'do his duty'.
Yes, your eldest is capable of 'watching out' for the younger three, but that's not his/her job and if something were to occur he/she isn't really old enough to do anything. Old enough to tell you afterwards, but not old enough to handle making a decision to call someone (your mum? SS?) for help or 'rescue' if it's needed. Besides, that's too big a burden to put on a 15 year old.
I seriously think you need to rethink your plans. IMHO, someone needs to be able to get to the children within a reasonable period of time if need be. And it appears that you don't have someone you can trust to do that for you.
Ask yourself how you would feel if he punched or verbally abused one of the children (and I can just imagine how well he's going to handle a wet bed every morning) and there was no one your children could turn to to get them away from him.
As far as him 'rewriting history' if you tell him the holiday is off, do you seriously think he isn't already doing that? Do you seriously think that he's going to sit them down and say "Well, I've been a bit of a shit father, haven't I?" or that he tells his friends "Good Lord, I was such a bastard of a husband, it's a wonder that New put up with me for as long as she did!"? Oh hell, no! He's already telling people that it's all your fault, and likely he'll tell the children that you have been preventing him from contact. So don't let that idea influence you.