I suppose after reading up on this thread, my question to the OP is now:
What would this relationship 'working out' look like to you? In two years, five years, ten years? Or do you view it as maybe a thing that will last a few years then naturally dissolve when the passion is gone?
The reason I ask is I think it will help clarify in your mind. From what I've read, I'm guessing that you would like to continue seeing him for 'long holidays' in his own country and maybe other travels in other countries, and continue a long distance relationship for the most part? Do you only want to go visit him? With or without your young adult kids? Do you want a baby with him? If you wanted to live together would you go over there and live with him? How would that work out?
Is he allowed to come to the UK at all on a holiday visa? Is he able to afford flights and travel to visit you in the UK? Would you permit him to stay at your home if he did?
Things to consider - are you sure he's not building up to using your address and your work / company as a sort of 'sponsorship' or statement of work viability for a visa application?
Also things to consider, that as soon as his feet hit British soil... in fact scratch that... as soon as he's on the flight over to the UK, even for a holiday visit, you know that he's going to be using every angle possible with everyone he meets to try to wangle residency of the UK and he may also already have contacts in the UK, plus migrant friends.
Now there's nothing wrong with that at all - everyone's entitled to better their life situation - but we must all be aware that for some bizarre reason, people from such countries think that coming to the UK is like winning the lottery and that we are all dripping with wealth that could help them, their families, and their entire village. I have seen this in action not just making some strange prejudiced assumptions.
It's one thing to ensure you don't marry him OK fair enough that's a great preventative measure -but- if he were to visit your home, what would you do to make sure he doesn't abuse that situation? What if he didn't want to leave?