Well, you do sound very very invested in each other, which is great. Could you go to some couples counselling together, to talk this through in a more supportive situation?
He probably needs to be a bit more realistic, and a bit more emotionally aware than just saying what he thinks you want to hear, which places a lot of burden on you, and doesn't help you with your feelings.
And it sounds like you need a bit of help to really track down what your specific worries are, and for him to support you around that. Like, if these children came into his life, would you feel partly sad about him turning his back on them? Because if he could to it to them could he do it to your children? Would you worry that he would resent you? See them behind your back? Love your own children less? Him just saying 'no none of this will happen, don't worry, it'll be fine, I'll do what you want' isn't going to help you feel better because it just doesn't feel like a good enough solution to the problem. But he's probably scared to say anything else in case he thinks it will make you leave. So you're both stuck in this difficult situation.
Sometimes couples therapy can help you both feel safe enough to talk about your deeper fears without ending up lost in a panic mode, and shutting down thinking. It does sound like there is a lot of goodness in your relationship, so it's worth doing what you can to work through this?
I think this organisation is good, it's London based but does webcam sessions too. tavistockrelationships.org/relationship-help/relationship-counselling-service
Or there are couples therapists everywhere really. Good luck.