Don't think he could stop you taking them abroad on holiday just because he has pr. He could apply to the court to try and stop you moving abroad permanently, but, even then, he may not be successful.
As the primary carer, I think you get to make the big decisions. All it entitles him to is to be consulted on various issues, to be kept informed of matters such as education, etc. Where there is a difference in opinion between you and him, then your views carry the greater weight as you are the primary carer. He can't use it to bully you into doing whatever he wants wrt the children. He might try but you just have to learn to ignore him.
What is holding up the payment of the money? Does he have it ready to hand over, but is withholding it until he gets what he wants? Or is he still waiting for it to come through from refinancing?
You could always say that you'll use his tactics as evidence that he is prepared to put his children's interests second to his own control-freakery, which may mean that he wouldn't get pr anyway. If the money is available, then get your solicitor to start the interest clock ticking.