Can it work? Yes absolutley.
The question are, do you want it to work and why don't you want this to happen yet or if at all?
You've hinted at your orevious husband being unfaithful which is a completely valid reason to be nervous or even recoil if the thought of taking the relationship to the next step.
Some of the comments in this thread don't appear to be helpful in the slightest and using this post as a platform to bash your partner.
At the end of the day, you know your partner better than anyone. Is he likely to be unfaithful? Is he the sort of person who wants you to meet his kids so he can leave you with them and he has fun whilst you have the kids?
If you want his to work, it will.
There hasn't been any mention of how soon your partner wants this all to happen. Is it a week, month or even years away?
It may be helpful to both of you to speak to your partner about your thoughts and concerns of his wishes and future plans and lay out what your future plans look like. Before this, it's important to question your own thoughts and fears and why you have these and understand them.
This is because if you want this to work, you'd want those thoughts and fears reassured and validated, right?
The only person who can do that is you partner.