I see a lot of posts where there's something fundamentally wrong or unfeasible about the relationship, but the poster always says how well they get on together and what a wonderful time they have together, etc.
I say it doesn't matter how well you get on together in the moment, there's a glaring incompatibility. So you get on well and have a great laugh. Big whoop. There'll be lots of other people you get on well with too. I've known loads of people, men and women, with whom I've got on like a whole city on fire. Didn't stop some of them being enormous see-you-next-Tuesdays at a later date. Because it's about who they are, not how well you get on today. This man resents your children and therefore has a mean, cold, stone for a heart.
He's not the one.
Plus he's blaming YOU for the past! He needs to be tossed out via a well-placed winkle-picker in the bum. Bloody nerve.
Also, why does he think he can say stuff about your kids to you like that? He's not only showing coldness and meanness towards children, he's showing a total lack of respect towards you. I suggest you let him know how wonderful you think your children are, that they're the best part of you, and how lucky any partner would be to have such amazing children in his life.
Bonus: He'll probably dump you and you won't have to bother!
If I had kids, and a partner didn't want them around, I would take that as a deep insult to me as well as to the kids.