So - a few things are important here - are you married? If so - you'll have to split assets (house, etc) 50/50.
As you are not mentioning selling your house - I am going to assume you are not married. Which, in your case, is a good thing.
Here is what I'd do in your case, before pulling the plug ->
Start start getting more involved in your kid's care. Do some more drop offs. Occasional pick-up. Work from home some time. Do what you can with other aspects of care - say medical appts. Move your mom in ASAP, so that day to day child care is not solely on him. Try to take some time in school holidays, etc.
When people separate - neither has an automatic right to keep the children. And initially it all relies on parents coming to an agreement of what they want/what's best for kids.
And only if they can't agree - you end up in court where a judge would decide how the kid's time should be split.
I don't know what your house ownership or tenancy is like.
So, if, for eg - it's all on your name - at some point you could ask him to move out. He'll then need to figure out his living and income situation, or stay with his parents. If at that point he demands kids stay with him most of the week and you only see them on weekends - you do NOT have to agree to that.
While all of that is going on - you'll be building up a track record of taking care of the kids - by yourself, by your mom, and maybe with a nanny. So - by the time he sets up his new household - he can, of course go to court and ask that kids spend more time with him - but it will be an uphill battle. As kids need stability and staying in family home is preferable.
If he is not able to adequately house them - it will also be hard to get even 50/50 for him.
So - in the end - you'll most likely come to some sort of agreement that makes sense for you and is best for your kids. You are not going to lose your kids.