Ok, in the nicest way possible...
If you ever want to find happiness in your life and a relationship that is really worth something, you won't find it while you still have contact with this man as he partly owns you.
While you still have any involvement with him, you won't be properly open to someone else, or even to the thought of meeting someone else as you'll always have a "maybe" tucked away.
He wants you there. He wants you there for his own uses - the shoulder to cry on, the one to confide in. But all while still with his wife, who is actually who he wants to be with, as demonstrated by the fact he's still with her. You only have his word about what is going on in their life / marriage and the word of a lying cheat isn't much really.
He's cheating you out of having a happy relationship elsewhere. He's cheating you out of a calm life without his drama. All while cheating his wife,
Stop letting him. Get some self respect and cut him off, totally and completely. Get a snail mail from him again - return it unopened. Do the right thing before he destroys your chance for happiness, just like he's already done to his wife.