He is being controlling and manipulative - abusive.
To clarify, you're not being stupid, you're not being a bitch, you're not being unreasonable, and you're not being a bad parent.
He is though.
You absolutely are allowed to tell people the reason you kicked him out - him forbidding you from doing so is another way to hold power over you and cause you prolonged suffering - and I would encourage you to do so. People won't be able to support you properly if they don't know what has really happened.
And if they don't know the truth it will be easier for him to invent his own version that makes you out to be the one at fault or worse.
The way he's behaving cannot be helping the children. It's distressing you (which they will pick up on) and it means they have no idea why they stand. Has he really left? Is he coming back? When will I see him?
I'm so sorry you are going through this, but there are things you can do to get control of your lives back.
I agree that you need formal contact arrangements. Have you looked into using a contact centre?
It's completely inappropriate and unacceptable for him to insist on coming inside your house for contact, moving things around, picking and choosing and changing when he will see them, and generally calling the shots in your lives.
His claim that the finances need sorting first seems like a way of delaying and controlling you. He is going to wear you down if you wait for that. You need to be the one to take charge.
He made his choices and it is not your responsibility to save him from them. You need to save yourself and the children. And to do that this situation has to change.
You need to see a solicitor, both to sort out where you stand financially and in terms of keeping him from swanning in and out of the house, and to formalise contact so that it happens at set times and it doesn't take place in your house.
The way it presently is, I wouldn't be surprised if your eldest is continually building her hopes up of seeing him and then having them crushed because it's all so unpredictable. If she knows exactly how many times a week and for how long she will see him it will give her some stability and reassurance. The younger one must be getting very confused.
He is behaving unreasonably and in a way that is detrimental to his children and to you - which in turn is no good for the children. This is not how a decent father would behave. Somebody needs to act in their best interests, as you have been trying to do.
So, take legal advice and get things set up properly. His current behaviour is abusive, so talking things through with Women's Aid might help too: 0808 2000 247. They will have more suggestions than us, and will be a real person reassuring you that you are not in the wrong here, and need to be able to protect your children from his behaviour.
Can you have an email address or phone number that is only for him to contact you about the children (perhaps change your existing ones so he only has the old ones, and let him be the only one to contact you on those). That way you only have to check them at set times and won't feel like you're at his beck and call.
Everything about your own situation should go through a solicitor. Anything about the children through those contact channels. And if he tries to use them for anything else do not respond or engage.
Don't try to reason with him. He's not interested in reason, he's interested in having power and control over you. You won't succeed in bringing him round, just aim for standing your ground.
I would consider having a set response to his various manipulations. So if he starts saying "it's up to mummy", you could calmly and firmly respond with "daddy knows he agreed he sees you on x days, so he won't be able to do that, but he will see you on x day, remember" ending on an upbeat tone and repeat ad nauseum.
If he starts calling you names, "that's unacceptable, you will not speak to me like that". Making threats about your parenting or preventing him seeing them, "I am protecting our children's best interests, you should want to support me with that".
If he's blaming you for what's happening, "you made your choices, these are the consequences."
Etc. But obviously use words and phrases that sound and feel like you. Just have your set, clear responses that maintain the boundaries you have set.
Everything you are trying to do is extremely reasonable, he is being outrageously unreasonable and causing all of you further harm and distress. That is not acceptable, and you are absolutely doing the right thing to put a stop to it.