money
You mention you have no money.
Do you have a job?
Is there a mortgage?
Is the house rented?
Do not move out yourself.
You have children he has abandoned. He has responsibilities which he will try to wriggle out of.
Please call a solicitor today for a free consultation to begin working through.
You are married - you and the children have rights. Start with that knowledge.
And investigate what benefits you may be entitled to.
record everything
Keep detailed records of everything. Write it ALL down.
Screenshot and keep every text, email, message, letter, whatever.
Journal every interaction, every in person or phone conversation. Recording phone calls. But better keep things in writing.
Note every instance of unreasonable or worrying behaviour - you're concerned about leaving him and his mother alone with the children, and you may need this.
Also note every time you've been reasonable, accomodating, offered visits etc as you mentioned earlier.
You'd be amazed how history can be rewritten to suit.