I think you need to push for no limits on calls to the absent parent and from the absent parent, within reason, and this should include a daily call if that's what the parent or children want. Additionally, the parent who is with them needs to not hover or listen to calls or quiz the children about topics discussed.
If your ex were divorcing in the US, he would be faced with this sort of clause in any custodial agreement, and if he objected, the judge would rightly suspect that he was using possession of the children as a weapon against you. The fact that he doesn't bother contacting them while you have them shows his game here. He's not actually interested in parenting. He just wants to avoid child support and keep you anxious.
I think agreeing to allow him to call the children and for the children to call him any time within reason (calls not to last hours and a maximum of one or two a day would be reasonable) with the same phone access for you would be perfectly reasonable. The children are young, after all.
I think you may have to give way over the three week trip to the US and every other Christmas. You could argue that the children's social lives, extracurriculars, and part-time jobs when theyre teens, would suffer, if this were to be ordered, and I think this is the best argument because the allocation of time with each parent has to be in the best interests of the children, not hugely disruptive to them, and not just a matter of the parent's convenience.
Arguing that they would be homesick or miserable is a criticism of ex's parenting and character, and would be dismissed by the judge because you have nothing solid to offer as evidence for this or evidence that this would be massively detrimental to them - and frankly, a father is going to get access to children even if he tries boiling them alive, in the presence of witnesses. Merely missing mum (though horrible for the children) would not be grounds for a judge to say no.
But he is being massively unreasonable in refusing to let the children contact you and you contact them when they're with him. Fight for the right of unlimited and private contact between the children and the other parent when they're apart. Sauce for the goose is sauce for the gander here. (And again, if he were to divorce in the US, he would have to accept this).