I'm sorry. This has clearly been a dynamic and defining connection for you, has given you the motivation and energy to change your life and free yourself of a marriage you had outgrown, and has been a source of unique pleasure. Which all makes it so very hard to let go of. But I'm afraid the only option is to accept and to attempt to forget.
He has done what he has done; he has always intended this to be temporary. He is anchored to his wife, and you have been a love affair. He has frequently tried to put the brakes on, and now you are totally free, he has drawn the line.
It sounds like you have been involved for some time, as you mention years. He is a fixture in your life. But he is still there. You can still have a lot of the interaction you had, I guess. The only two issues to me are that it's hard to know him in a new way, where you have effectively been rejected/terminated, and your feelings for him subordinated to his feelings for his home and wife. This is always going to be a problem when you get involved with someone who already has a home and partner, as I'm sure you knew, but knowing is one thing and experiencing it is quite another.
The other issue I see is that he is now saying he regrets having gone on the site, opened up his life and himself like that, and had this encounter with you. That seems emotionally ungenerous to me, but he may be doing that to try to push you away further and minimise your connection. It does sound a bit as if all along you were falling for him while he was only one foot in, never wanting to fully commit, because of his existing life situation.
You need to make some facts very clear for yourself, and accept them, however long it takes. Repeat them as often as necessary.
He cannot give you what you want.
He may have released you because he knows he can't do that, and it would be unfair now you are free only to give you occasional intimacy.
He is not going to change on this.
You deserve someone who loves only you, and to be the only person in his life.
It is actually pretty mean of him to say he wishes you'd never met.
He wishes you'd never met.
He doesn't want to be with you.
He will be your friend at a bit of a distance.
To me, once you accept all that, you will not want to be around him.
You will want to find someone else.
I don't think there's anything wrong with talking to various people online until you find one you like.
Just make sure he is free, then you'll be free to fall in love with him.