I would also add a few words of caution here. Not sure what the personal circumstances of the other man are? That does make a difference.
If you get on well with your partner then probably what is missing is the romantic love/intense sexual side of things that you (hopefully!!) get at the beginning of a relationship. In general I think this tends to wane as the years go by - although I know there are some posters on here who still claim to feel like this in very long term relationship - all I can say is that I am very envious of them!
But I think quite a few people end up in situations like your own and it is very easy to develop a crush. A similar thing happened to me. I convinced myself that my paramour was madly in love with me (in reality I think he liked the challenge of trying to seduce a married women - nice, eh?)
Nothing physical ever happened and, in time, the feelings passed. I do not regret it as it made both me and my partner make more effort in our relationship. However I realise that the OM was a player who never really cared about me, despite all the talk. It was all about him, his ego and his d*.
With hindsight, he behaved very badly indeed - deliberately played on the emotional side of my nature to seduce me in order to try to have a sexual relationship, knowing that there were big risks (on my side).
Worse than that than that he enlisted the help of 'friends' (his not mine) who egged him on in order to get vicarious kicks out of it. And of course I and my family became the subject of gossip and speculation which was horrible and made me feel very cheap.
Once the feelings had subsided I realised I had been made a complete fool of, subjected my partner and children to malicious gossip and in general been 'played'.
Yes, I take responsibility for allowing the crush to develop and being naive enough to think it would just be an exciting romance that would spice up my life. I have a million times more respect for my partner, who weathered the storm, while the OM, as far as I am concerned, is a little sh** and so are his so-called friends.
Anyway, what I am saying is be careful because there are some very nasty people out there who are higly manipulative and it sounds as though what you have at home is pretty good.