Bethechange - speaking from experience, I would strongly advise you as the resident parent NOT to go to court. Keep things out of court, unless you have compelling reasons and absolutely have to go ie. abduction, or abuse. See the many threads on here about the way women have been overridden by the courts. Sorry to be blunt, but the courts will NOT grant residency on him being an hour late. They will be unlikely to prevent contact due to no heating. So you are wasting your time, and yet they will formalise something that you will then have to abide by.
I did have compelling reasons, yet it was still very much biased towards the father. Read the many appals threads on here and you will see the horror stories other women have had in court.
You will not get residency unless he has abducted the children. I had strong reasons to apply for residency but still they didnt give it.
You said yourself - you are not sure what you are going for, so realistically in that case you will be unlikely to convince a judge! It is not easy.
You really don't need to go to court. Simply send him an email stipulating the time they are to be returned. If he does not return them or answer phone etc call the police.
If you are not happy with the accommodation, put it in an email. Ask him if there is somewhere else he can take them. Would you be able to offer him your place (then you know they'll be back)?
And if you go on holiday simply inform him, by email, that you are taking them to wherever on whenever. If he has PR you do need to inform him.
If he doesn't like these arrangements, let him take you to court. You will have the email evidence that you have been very reasonable and accommodating.
Keep it away from court as long as you can. You really will not do yourself, or your children, any favours if you don't. Be strong, be firm, be assertive and be reasonable. But don't let him bully you! Best of luck.