Several years ago, I worked closely with someone on a project I was consulting on. I'd met him before professionally, when we were both married, but not thought much about him other than that he was very good looking. A few years later, we ended up working on this very intense project. He was divorced by then, also smart and funny and creative and hardworking and sexy and I got the sense he saw me the same way. I thought about him a lot, like a major teenaged crush. When the project wrapped, we all went out for a big celebratory dinner (both away from home) and afterwards, he told me that he didn't want to mess up anything in my life, but was very attracted to me and just wanted to let me know.
When it went from being in my head to possibly being something real, I knew I needed to step away, so I said something like thanks for telling me. I didn't want to say anything similar back because that felt like it would be a betrayal of my husband. I went home and put some real energy into my marriage, which was good, but definitely sighing a little under the stress of 3 kids, 2 dogs and 2 pretty intense careers. I'm not saying the feelings went away overnight, but they did go. I've seen him since and still think he's very attractive, but didn't feel that same pull.
Something that really helped was trying to look at my husband through different eyes - as though it was him and another woman in the situation I was in. I'm positive quite a few women over the years have also seen him as smart and funny and creative and successful and hardworking and sexy, all of which he is. He's also a great husband and father, which I suspect the other guy might not be, or he most likely wouldn't have ended up divorced.
One thing I would say @80schild1 is that I think masturbating over him while not having sex with your husband is edging dangerously close to an emotional affair. I don't think there's anything terrible in general about having sexual thoughts about other people, but in your shoes, I'd shut that down.
You can float along on circumstances but after a while it starts to become about making choices and decisions.