If you don’t want them taken abroad, you HAVE to do something preemptively.
My ex took my children abroad without my permission, outside of the time he should even have had them. There’s been lots of police involvement with violence between he and his GF, and her towards my children, so I was justified in having concerns.
By law, a child can not be taken out of the UK without the permission of both parents. In practice, this means nothing. He’d stolen their passports months before, denied it so I cancelled them. He was then able to get out of the country on those same ‘lost’ passports, despite me speaking to the airport he was flying from directly, despite me speaking to the police based at that airport, and letting them all know my children did not have maternal permission to leave, and were not travelling on valid passports. All fell on deaf ears, and passports for UK citizens are apparently only visually checked when a UK citizen is leaving, so not against a system to see if valid.
If their father believes he should be trusted to take them on holiday, he can apply to court to have them grant permission and overrule you, but his actions will be given a very grave view by a judge/Sheriff if he removes them without your consent.
There’s also a team called the Caveats team at your passport office. They can put a flag against your children that you do not consent to applications/changes made by the father. Hard to find their details, I’ll warn you, though.
Hope this helps.