If you look at the UK government guidelines, you'll notice that they are suspiciously vague. so it specifically says that you need the permission of the other parent to take a child out of the country, but does not provide any details on what that permission might look like.
In REAL life, what happens is that any parent travelling without the other parent takes birth certificates as well as passports. There is a decent chance that said parent will be asked more questions than other people at immigration, but usually this appears to check that everything is fine, the children are travelling happily etc. I have been asked why my child's dad was not travelling with us once (but was not asked to prove that he was okay with it) and have been asked to prove my relationship to the children on a couple of occasions by producing a birth certificate (they have different surname to me and while we all have UK passports, I was born somewhere else so I suspect that makes it more of a flag. Especially with DD who does look like me, sort of, but has her father's dark hair, skin and eyes vs my red haired/ pale skin/blue eyes).
theoretically, if you have a volatile relationship with an ex, he could pre-empt your travel by putting a flag on you as traveling without permission. I have never heard of this actually happening, but I absolutely can believe it. But it does not sound like this is an issue in your case.
On MN, people are always terrified of this. In reality, in the very very unlikely event you're pulled over, they'd just call him.
edited to add: You should, however, ALSo check the rules of the country you're travelling to. As a PP says, South Africa, for example, has stricter rules. New Zealand as well I believe. POssibly the US.