Let him make the application to the Court - it takes the power away from him and into the hands of the Judge or Magistrates. That's much better for you.
CAFCASS will do safeguarding checks, which will reveal any previous police involvement etc. Also disclose the Non-Mol in place.
It's likely the Court will order a safeguarding report (either a s7 or a s37), and all this will take time. Given the distance between you now too, it's not going to be an easy matter for him, which is good for you.
If he thinks he's going to rock up at Court at the first hearing and get what he wants, he's got another thing coming. The Court can't make an Order for contact at the first hearing unless there's an agreement, which obviously there won't be.
The Court won't consider contact until all the safeguarding work is done.
Honestly OP, you have much more of the power once it's in Court. If he makes the application and there's an unfavourable s7 Report, or if he loses interest half way through the proceedings having not immediately got what he wants, then you will have all that to support your position if he ever gives you problems again.
Hopefully you have decent legal aid availability in your area. Look up the requirements on the LAA website and make sure you have all your documents ready for your appointment with a legal aid solicitor.
If he issues in his local Court, you should apply to have the proceedings transferred to your local Court. The Court will transfer, because the rules say it should be the nearest hearing centre to the child.
It's scary, but I'd rather be representing you than him. You have the better case. Sometimes the Court getting involved is a good thing. He may not tolerate not getting his own way and he may not be willing to "jump through hoops" or engage with professionals. Let him show himself up, which is inevitable.
Good luck OP.