my 4 children spent a week at easter with their father, I didnt know or ask where they were in advance, and they returned filthy, "some time on Friday". (age 3 - 10)
depends what you mean by dont trust. If he is the sort of person who would drive with them while drunk, then dont let them go, of course it is true that children dont need 2 parents at any cost.
But if you mean you cant trust him to provide care to the same standards as you, then you should let them go IMO, because they are his children too. Sorry if that's not what you want to hear - I know that this is a very sensitive subject. Only you really know why you dont want this visit.