Tell him to fuck off. That is your and your children's home, unless he wants to pay for another one for all of you. Does he seriously just think he can tell you to leave? What does he think you will do, walk out without your children leaving them with him?! Tell him if he's that unhappy, he can be the one to go. He knows where the door is!
Don't you go anywhere. Tell him if he can't stand to be around you, he will need to go as with three kids to care for you will need the space. Tell him that this is his issue to sort out. You don't need to run around making everything okay for him right now.