OP you're not going to get too many handwringing, "poor you" replies here, you'll get straight-talking advice which, whilst difficult for you to hear, is well intentioned and given in the hope you'll do the right thing. You don't need tea and sympathy, you need the harsh reality of what you're doing to be pointed out to you.
Your man is an absolute cunt (sorry for those of you who don't like swearing). Would you want him anyway if he left his wife? Could you trust him? He's a cliche, you're a cliche. You can tell yourself you didn't do this with malicious intent, but you have, so grow up and take responsibility, you've knowingly slept with a married man, I'm guessing this is not your finest decision in life, but it's a decision you actively made. Put yourself in his wife's position, how would you feel? I bet you haven't always used protection either have you?
You are stopping yourself from finding someone who really loves you. I'd hate to share a man with anyone, why do you accept this (or does he tell you he doesn't have sex with his wife 
)?
End it, go no contact. If you can't do it for the wife do it for yourself, you're wasting your life on a man who lies and cheats his way through life. Any woman deserves better, including you.