I don't think you should leave, not at all.
Why should he - the one causing the problems, the one shirking his place in the family, the one who causes dissent and bad feeling, the one who refuses to help out ANY of the other people in the family (not just you) - why should HE have the right to stay, to not be disrupted?
The children come first and they should get to stay in their home. Seeing as he cannot even be bothered to make sure they are cared for and safe in an emergency (!) I assume that he would not be regarded as the primary carer, so you should stay with them.
Incidentally, it's also common sense, financially too, for one person to have to move out and pay for alternative accommodation rather than several.
No, of course he won't want to go and doesn't want the marriage to end. Why on earth should he? He's comfortable. He gets his domestic needs met. He gets to be the 'part of a family' man that society expects, rather than a live-in-a-bedsit-oh-dear-what-went-wrong-there man, a bloke getting awkward questions from everyone from his mother to his work colleagues. And he is sufficiently thick-skinned in his nastiness to not feel that he has to at least respect or acknowledge his family if he wants to live there, so it costs him nothing.
No, you will have a big job getting him out. The first stop there would be a solicitor. The second, telling friends and family that you are split, and why, and that he is refusing to move out. The third, stopping all domestic aid for him and living as if he is not there. You would have to make him very uncomfortable.
You spoke of counselling and depression and whether he is unhappy. I don't know. Yes of course he could be - you know him - what is your gut feeling? Mine is certainly that the extent of his deliberate nastiness towards not only you but also his children takes this beyond the level at which I would be considering his feelings any more. He is vindictive. You say you know he was reading your messages - fgs, what was going through his head there? Indifference? Pleasure that you were having a hard time? What the hell did he think would happen about the children? (Actually I know the answer to that - exactly what did happen - you would arrange for the problem to get solved. He already knows you don't rely on him, and he likes it and makes no attempt to change it.) Love messages all day, copied from the internet, eh? Self-serving little shit.
If it were me, I don't think I would bother with counselling, because I wouldn't want to be with this person any more, and would by now believe that they were basically the person they were - unpleasant. You seem to be there too. I don't think you believe that he can be 'fixed'. And you DID leave, and all he did was to pretend to you until you came back. He KNOWS how you feel, he just does not care except to make sure that you stay there and his life is not disrupted - as you say, he denies that there's a problem. You could speak to his dad - but again, what is it that you want to achieve? You see, for me it would actually be his attitude towards the children which would destroy it. He may be so thoroughly unhappy with you that he feels utterly vicious towards you - but his nastiness to the children is much harder to excuse. They feel it already - you say they would rather not be with him. That is a big, big problem which will get worse.
Is he having an affair? I don't know - all I do know is that if this were my husband, I wouldn't care if he were by now - I'd probably shake her by the hand 