This is just awful, I never understand how some men (mostly) can just turn on the mother of their children and act this way, if you want out just get out, no need to emotionally abuse someone.
So first things first, you're in a horrible situation and it'll be shit for a while but he's shown you his true colours, you're better off without him, you deserve better.
I would be first of all transferring half of whatever is in your joint savings to yourself, if he wants to piss his savings up the wall in the pub and kebab shop let him crack on, but no reason he should have access to what is half yours as well, you are entitled to it, it's half your money, don't let him think he can control you in that manner.
Next would be telling him that at some point the kids probably will want to see him, you need to know where he is staying when that happens, if he refuses to tell you where he is staying then the kids don't go to him, end of story.
And please take control of this and start divorce proceedings yourself, I would have a fairly confident guess that he's been cheating but sounds like he's a coward that will never admit to it and is emotionally manipulating, abusing and gaslighting you in his attempt to get away with it. I once wasted more time than I care to think about on a man like this, he never admitted to any of his lies even in the face of irrefutable proof, that's not a normal way to behave, you and your children deserve better, don't let them grow up thinking the way their dad treats their mum is normal and acceptable.