This if from Jackw's link above: "If a father knows he’s found someone he can trust around his kids and is certain they will be present in his life for a long time, most experts recommend waiting at least six months before coordinating a meeting between children and the new partner.
Nancy Fagan, divorce consultant and owner of San Diego’s Divorce Help Clinic, says that six months is essential, but it must be six months of exclusive dating. For some families the time may be longer.
“If any of the children are still in pain over the separation or divorce, dads will need to wait longer,” Fagan says. This is to eliminate confusion while kids process their pain and grieve the loss of their former family unit.
Other situations prompt more time. Fagan stresses new partners who happen to be friends with the ex-wife, have a significant age difference, or are the first partner after a divorce are all very likely to upset children and the father’s former spouse. Relationships that share any of these features, more often than not, do not last."
I suggest you quote some of that to H. Though perhaps not the last sentence.
I still don't think you should feel obliged to do what he tells you to, I know you want your children to see their father, and you need some respite, but, unless you prefer it, I don't think they should stay away from you with him overnight at the week-end, yet. I have nothing official to base that on, it is just my gut-feeling in your circs.