Stay where you are. Although if I were you, I would have had all the bedclothes off him and sent him to the sofa.
Do NOT leave the house. Why should you, and why should your children be pushed out of their home? They will be completely bewildered, frightened, upset - this doesn't need to be the case.
If he has a mother, tell him to go and stay at hers for a while. If he doesn't, then he can go to a hostel/hotel/B&B/friend's house.
I don't blame him for refusing to leave in the middle of the night, but he might reconsider in the morning - if you give him enough of a shove!
If he still refuses, start by getting a suitcase/bag out for him and if he really won't shift, you might have to pack stuff for him. Pack what he's likely to really need this coming week and then chuck it outside. Once he's out the door, bolt it and refuse to let him back in.
Get him out of the house by hook or by crook so that you can have some time to clear your head and think things through - stay in contact though, just make it on your terms only.
Also, if he refuses to leave, it is likely because he thinks that what he's been doing "isn't that bad" - you need to make it VERY clear to him just how bad it is in your eyes, how much of a betrayal, how hurt you are etc. etc. If he's halfway decent that should be enough for him to realise you need him to leave and he should go voluntarily - if he still refuses, or refuses to accept he's done anything wrong, then you have even more serious problems ahead. :(