I’ve told him how I feel regards to his kids but he says he will do what he can to make this work.
Except he won't do everything he can.
Because if he actually listened to you, he'd say yes sorry it's moved too fast and I get that it's been too much, too soon, let's go back to dating again rather than whole weekends together etc.
Instead he's just kind of shushed you and made you feel like you owe him a relationship because he's (apparently) not an arsehole.
In actual fact he sounds manipulative and emotionally incredibly immature.
You're worried about breaking up with someone you've been with for only five months as he's said or made you feel like he 'can't live without you'?!
We'll you'd hope his children, who he apparently cares about a lot, would keep him hanging on and getting on with his life really wouldn't you?
Your boundaries are way off here I'm afraid. You met him at a time you felt very vulnerable emotionally and it must have felt nice being wanted and complimented, having someone to talk to etc.
But the relationship has moved at a completely unhealthy pace, especially considering children are involved.
Him bringing kids to your home this early on is so irresponsible and I'm afraid you letting him do so, and thinking it's sensible, isn't a great decision either.