Family lawyer here.
The law is not drafted in terms of "rights", but in terms of responsibilities, but it boils down to the same thing.
What gives you the right to make decisions about a child's upbringing, medical treatment etc is called parental responsibility (PR).
There is legally no means of removing PR from a father who was married to the mother (conversely, you can legally remove PR from an unmarried father) - unless there is an adoption/surrogacy. But you can get a court order to limit his ability to exercise it (which is as good as removing it). However, the court would only make an order in truly exceptional circumstances (eg where there had been sexual assault or violence by F, where F has refused to consent to significant medical treatment). I do not think a F saying he doesn't want to see the DCs is going to be enough.
If you can't remove/limit his PR, then technically he is entitled to make decisions about the DCs - where they are educated, whether they are brought up in a particular faith, where they live, what medical treatment they are given. In practice, since he has said he intends to have no contact, just make decisions yourself and don't consult him. If he ever tries to interfere in the future, and you can't agree, then either one of you could apply to the court for a specific issue order dealing with whatever the issue is. Where a F has had limited/no involvement, his wishes are extremely unlikely to be preferred over the mother's.
If you are more concerned about your ability to take them abroad on holiday without his consent, then you need a residence order. This gives you the right to take them for up to 28 days without his consent. You could ask him to consent to a residence order. This will protect you in other ways (eg if he suddenly decides he wants to play a role, things turn ugly and he removes the DCs from you).
In practice, you don't have to have a residence order - you could just go on holiday, without asking him to consent. Technically this is a child abduction under special international legislation (Hague Convention) and he could apply to get you sent back, but if all you do is go on holiday then you'd be back before the proceedings got underway and so in practice the risk of this happening is negligible.
If you move abroad permanently, then again technically you need his consent and if you don't get it you could be forced to return (whether or not you would be is fact-specific).
On balance, I'd say get a residence order and then just make parenting decisions without consulting him, since he's made it clear he wants to disengage. What he said to the solicitor must be documented/formally recorded in some way (eg in a letter).
You cannot protect against him deciding later down the line that he regrets his decision and wants to see them. Again, what will then happen is fact specific and will depend on things like when he last saw them, their own wishes, any threat to their welfare etc. He will be expected to prove his commitment.
If he really does this, be ready for the children to "romanticise" the idea of the absent father when they are teenagers and rebelling against Mum's rules.
This is a good article to read, which explains it all clearly:
www.familylawgroup.co.uk/site/blog/flg-news/when-can-parental-responsibility-be-removed