Ask other local residents who are from other countries who have been through similar situations for recommendations for lawyers. You want someone who is totally on your side. You would be surprised how many lawyers see the needs of the father ahead of the child.
Get all of your evidence in order; proof of abuse, proof that he consented to the move, proof that you have facilitated contact and agreed to his requests despite your child's reluctance to see him, proof that he has not wanted more access, did not provide primary parent care responsibilities before the split, get a neutral party to interview the children and ask them to state their wishes as part of an evidence pack, this could be a lawyer or if they have any external support like counsellors etc.
Once you have all of your evidence and your legal support in place then wait.. and see what he brings to you. It might be an empty threat. He might not be willing to engage the law to pursue this. If he brings it up, ask him to address this to your lawyer. If he makes changes to the contact arrangements, remind him that you have an agreement. If the contact agreement is not in writing, not written in the presence of lawyers or not court agreed, consider whether this is something you would wish to pursue.
Once he brings a case, my understanding is that there is a commitment to get it resolved within 6 weeks, so you will not have much time to get things in order. However, in advance of this happening, keep documentation that you have made every effort to meet the needs of your children while facilitating contact within the existing agreed terms.
Good luck, get your support around you.. if you are in Switzerland, I can put you in touch with people who might be able to help.