Wendy, I have a lot of sympathy for you. I would love to say that it will all work out the way you want. Unfortunately I'm not sure it will. Let me try to explain how the courts would look at this.
Their starting point is that your children are entitled to contact with their father. It is important to understand that this is the way the courts look at it. It isn't about his right to contact with them, it is about their right to contact with him.
The courts take the view that in most situations you cannot control what he does or who he sees when he has the children. Equally he cannot control what you do or who you see when you have the children. There are limits to this, of course.
If I understand you correctly there is no contact order in place. That means you are free to stop contact. However, if you do he is likely to use that as evidence that you are unreasonable. You may then face an uphill battle to convince the court that you had good reasons for your actions.
He has been violent to you. However, as I understand it he has not been violent to your children nor has he sexually abused them. It is possible the court will order supervised contact but I'm afraid that in these situations they frequently order unsupervised contact. The fact that you have allowed unsupervised contact up to now may suggest to the court that it is ok to continue with that.
The courts frequently come across mothers trying to stop their children having any contact with their ex's new girlfriend. The fact that you have allowed unsupervised contact until she came on the scene may lead the court to conclude that you are such a mother. And I'm afraid you have no evidence to justify barring her from contact. All you have is a statement by your ex which he will undoubtedly deny. Even if you were able to prove he made the statement that doesn't mean the violence took place. It may have been an excuse or he may have been being a drama queen. Whilst you can go for a prohibited steps order to try and block any contact with her, in my view you are unlikely to get it and it may lead the court to believe you are being unreasonable.
I think the best you can hope for is supervised contact but you should be aware that there is a reasonable chance he will get unsupervised contact if this goes to court. You definitely need to see a solicitor who specialises in family law. If you hunt around you should be able to find one who will give you a half hour consultation for free.