I left my ex not because of, but triggered by another man.
Why I really left: the "ick". I'd been having it for a couple of years. Then thought a baby would fix it. Which it didn't (of course ... how naive of me).
What the OM did: nothing, really. He was just a crush - secret and, as far as I know (never told him/asked) unrequited. But his mere existence reminded me that there might be something better out there than grinding my teeth at the fact that my OH, whom I resented for even daring to exist at that point, until I bite the dust.
Do I regret leaving? No, not at all! I'm far happier now. Casually dating, nothing serious, but it's most definitely an upgrade from living with someone you've come to be utterly repulsed by out of sheer duty/because I felt it was "the right thing to do" and I had "no tangible reason to leave".
Should you leave over another man? No! He might not be worth it/you might tire of him or he of you/it may not work out for any number of reasons.
Should you leave over "you"? Yes, absolutely, if you can't answer with a decisive and enthusiastic "yes" to whether or not you're looking forward to spending the rest of your life with your partner - regardless of any ongoing affairs/crushes/etc. I'm not into the whole "relationships are hard work" philosophy. They can be, for a short period. But if the times when they are far outnumber the times when you're a happier person because you're in them, it might not be a relationship worth working so much for.