I would also agree with not letting them be alone with him.
Remember, just because they are family it doesn't give them any right to bully or emotionally blackmail you. You also are under no obligation to see them or let your children see them.
They don't sound like nice people anyway.
Anyway, come back and let us know how you are.