Have you offered their dad some kind of contact in the mean time? Through a contact centre is one way to do it.
If he is serious about seeing the dc he will agree to it.
By stopping contact completely you will look unreasonable in court I'm afraid. Even if you feel your reasons are justified there are ways (such as contact centre) for contact with dad to continue.
The same argument will be used on you, you are not a relative and may well have had the dc in your bed alone or bathed them...?
I am not saying it's right or wrong or that the circumstances are the same. I'm just saying look at it from the court's point of view and how it will be put to them. Thinking the worst isn't the same as it being true. You can't accuse this man without proof. But you can ensure the dc aren't put into a situation where they could be at risk.
If SS and the police are saying there's nothing they can do then court and CAFCASS (who spend far less time coming to a conclusion) will say the same more than likely.
SO look at alternative contact arrangements with tehir dad in the mean time, and see if their mum can speak to him, maybe with teh help of a mediator, to explain why you find it inappropriate and to work on solutions.
At the end of the day a court may order contact regardless and you'd have to send the dc or be in breach of the order.
Far better to offer something on your terms and know the dc are safe in a contact centre for the time being while making your DP look like she is being reasonable to the judge should/when it comes to court.