Ex wasn't able to care for our children safely either.
At first we met in a cafe for tea so he could see them on neutral ground, and I did supervising.
Later I took them to his house... He got to see them, but I supervised. I ended up cooking a lot as it was best for the children.
He came to the flat while he could behave, but got banned when he was an arse and threatening. He was better on his territory.
I supervised contact out and about, and at his house for several years as he was not safe with them. I gradually pulled back and sat in he garden having a coffee. Once they were old enough to be safe with a little bit of oversight, they went one at a time. Then he learned to have them both together. They now go away to his family and come and go to his house with more choice on their part.
It was fucking hard, but my main aim was to keep the children safe.
I suggest that you meet him out and about at the weekend for a bit until you can establish a bit of separation, ( soft play, park, cafes)
After a bit consider supervising in one home or another.
Has he done much parenting? Had them on his own? Work out what he is capable of, now, work towards him doing more. Eg can he be trusted with them out and about? Is it just falling asleep at home that is the trouble?
What works for one family may not work for another.
Lots of people say he will learn to look after them... Ex was only capable of that when they were older. I had seen enough to know he did not learn and it was fucking dangerous. ( Third floor big window open with climbing toddlers?) Letting go of hands and letting them run off? ( Autistic bolters)