What you're suggesting is the ONLY sensible thing to do in current circumstances.
People like your ex can bleat all they want about it being unfair, or what his 'rights' are but these are extraordinary times and health comes before anything. Even maintaining face to face contact with non resident parents. Unless absolutely necessary, and this isn't, we keep our distance.
You have no idea who or where or what safety measures your ex is taking so exposing your child to a.n.other person, even if it's their dad is yet more risk. If he wants to fuss about it and, for example, threaten you with court for not adhering to the order, so what. He won't get a hearing until God knows when and no judge in the land would damn you for practising responsible distancing.
Of course, he can call, write, text, Skype etc.
I say all this as a stepmum whose partner's ex spent over a decade messing about with contact, using the children as weapons, obstructing CAFCASS as was etc. My partner was almost driven to a breakdown by the situation and being kept from his kids..... but that's a sacrifice he'd have made without objection to keep them as safe as possible, and to stop the spread of this virus. No question.