I dont have time to read all the replies so apologies if anyone else has said this already.
Here is what I would do.
I would try and talk one last time, in an ultimatumy way, possibly farm out the kids beforehand and make it clear you will not be fobbed off and are about to deliver an ultimatum.
If he flat out refuses to talk to you, say "okay then, I'm going to be moving this to the next level in due course then" and finish the conversation there.
I would take a day off work, when he is at work, get the locks changed and pack him a bag. Basically I would put him out. I would pre-book him a couple of nights in a hotel so he's not literally sleeping on the street.
If this made him talk, fine, but it would have to be on neutral ground somewhere, and proper talking. I would ask him to move out and have a trial separation. If he agreed to counselling, fine, I'd do that. In the meantime he can't come back in the house, etc. The kids would stay with me in the meantime unless they wanted to go with him wherever he was, but that would be their choice.
If he wouldn't talk and would'nt compromise, you basically carry on in the house, and initiate divorce proceedings like you would if he'd cheated on you, or hit you, except his unreasonable behaviour is that he doesn't talk to you and has effectively opted out of the relationship.
If in the process of divorcing him, it comes about that the children want to go with him, then that may come about - but might not. It might be that you be the one who has to get a flat in the end and that you do get joint custody or it might not. You just might be surprised.
If you do it that way and what you think will ultimately happen does happen, then you are only as bad off as you would have been if you'd just done it, but it wont be as much "your fault". Your children wont see it as you leaving them - they will see it as putting their father out but they are old enough to have a bit explained to them, you can say to your six year old "mummy and daddy need to talk and mummy can't live with daddy until he talks about problems" and to your daughter you can explain a bit more.
In the meantime you can create a stable environment for them in the family home and take it from there.
That is what I would do, off the top of my head.
YANBU to want out of this relationship.