I can understand your concerns at a time where there are so many unknowns. Firstly if there are any court orders defining contact then these must not be breached and if you decide to restrict contact in the face of such orders you need to take legal advice. If you have to reduce risk to the children or others then the court is more likely to be understanding if you act without recourse to the court first. But this should be discussed with a solicitor unless there is no time or it is not a situation due to urgency where you were able to.
I think it is a matter of risk assessment and your relationship with your ex. There is minimal risk to young children according to the statistics but there is always some risk and there are concerning cases that are now emerging showing that children may be at risk if they are obese or have an underlying illness.
There are also warnings about using aspirin, ibuprofen, Nurofen or NSAIDs and if your children need to take these due to illness then you need to talk to GP or NHS direct.
Children whilst having a low risk of developing the disease may still carry the disease and if they then have contact with elderly people such as your parent's grandparents above the age of 50 then the risks for them are greater as they infect them and they are more at risk of having complications.
I presume your ex is understanding and has his children's best interests at heart and I think that you should discuss these risks which are greater if he has to meet a lot of people each day.
Unfortunately, there is no way of assessing the unknown, children are more likely to carry the virus but not develop the disease and therefore they may not show symptoms. This means you have to assess the risk to others they pose. The govt has now asked that all high-risk people to cut themselves off and this will reduce the risk it means that your children will not be able to see grandparents etc for 12 weeks min anyway. On top of this your children if you cut them off from their father face the prospect of not seeing their father for up to 18 months.
It's a very difficult sad time and all I can say is that there is no real answer you as a parent have to make a decision hopefully with your ex-partner's input. As a parent and a solicitor, all I can say is that it is normally in the interests of your children to have a relationship with both parents and disagreements between the parents can be damaging but there are different times and as the main parent you have to weigh things up and discuss with your ex. I think you should see how the land lies and then come back to the users of the forum for the great support they provide you in hopefully helping in this dire situation.