OP - stepping back a bit, you aren't unusual to feel this way once a married man is going to actually leave his wife and it all becomes real - I have a friend who's been the OW twice (that i know of) - both times when it looked like she was really going to "win" she went off them.
She both times has different reasons she gave to her friends, but really it was a combination of enjoying the thrill of the chase/the challenge of pulling a man she couldn't have (particuarly seemed to enjoy making him give up family time for her), and not actually wanting a commited relationship - she will argue otherwise, but never really wants to do the living together, "who's turn is it to put the bins out?" "can you put your dirty pants in the basket!" type relationship - her last one that lasted a long time was with a man who lived at the other end of the country (but unmarried - thankfully) and then she lost interest once he'd started getting job interviews in her town and talking about moving in with her. (I stepped away from her a few years ago and don't know her most recent romantic dramas - we're facebook friends only and she's officially 'single' on there, so could well be the OW again).
First things first, make it clear to him that even if he leaves his DW, you don't want to be together - be very, very, very clear you aren't just testing him, it's over if he leaves her or not. Then cut contact, let him decide what he wants to do.
After that, you might want to assess why you felt the need to go for a man you couldn't have. It obviously wasn't that he's the love of your life, so was it that you were avoiding a relationship that could lead to proper commitment, or that you were enjoying the 'naughtiness', the chase of getting a man you really couldn't have etc... best you look at that before you think about dating anyone else.