Right. You need to walk away now.
I've told 3 married men that I'm not interested this summer alone.
I'm nothing special, in fact no single man is ever interested in me, but men who fancy a bit on the side, yeah, bring 'em on, they're all after me... 
One of them, in particular, was a real shock. A lovely, decent man, all of that, hinted that he had fallen for me and all of that nonsense. And I told him I wasn't interested. That I was worth more than being a married man's bit on the side and that his wife deserved better to.
Haven't heard from him since. Well I have, but only to be sure I was serious and tell me that his feelings were genuine and that he was working out how to leave his wife so that we could be together. Etc. I told him to go away, sort out his life and come back to me when he was single.
Do you know when I expect to hear from him again...? Never.
Why? Because if he was really that unhappy with his wife, he'd already be gone. I was targeted by my first married man when I was 19 and have been hit on by them periodically since. It's not flattering and it shouldn't be flattering to you. You shouldn't even be thinking of this as a relationship or a fledgling/potential relationship because, at the moment, you are just being a bit of a shit and a fool to boot.