You all need consistency and routine. Not guessing when he is putting in an appearance and you are not sitting around waiting for him to grace you with a visit.
Set time and days. How ever that looks. The children are young then need to know when and where they are seeing him. And you need to be able to grieve in peace.
He has left the family home. He made a choice it is now not for him to come and go as he pleases.
I know how terribly hard this is. You need to dig deep now and work a plan that is in the children's best interest.
Seek legal advice. Gather support around you from friends and family. Gp if you feel it is appropriate and counselling to get it all out. Journaling helped me
Communication is now solely about the children.
Money through CMS
Contact is on x and y day
House etc through the solicitor.
Show him you won't be pushed around. He will shout. Get angry and stamp his feet. Let him. He doesn't get to dictate to you anymore. It is a tantrum because you are taking control of the situation.
I promise you it gets easier in time. But for now you need to dig deep. Be kind to yourself do your grieving and build a new life for yourself. It's not what you wanted or planned but you will come out the otherwise in time
And your children will be fine. They need loving strong parents and you have got this in spades.