OP, been there, done that (including the shagging).
What you need to do is think, really carefully, about what you want.
That kind of attraction is like a drug and is enormously exciting. The question is: what is your ultimate aim? Do you want to leave your husband? Does he want to leave his wife? Do either of you or both of you have children? You say you're "happily married" - so this is decision time. If your marriage really is happy, is it worth unseating it? And if you really think it's worth unseating it, is it really that happy?
If you have an affair, it's likely to wreck your marriage. If you don't have an affair, you will go to your grave with your secret knowledge that there was someone whom you were mad about. The feeling will pass, btw, if you let it run its course. I had a couple of these before I did have an affair; the 'run its course' ones were very intense while they lasted, but never went beyond flirtation (though I sort of regret that now).
If you want to remain married (and it's worth asking yourself if you really do, and why), then you basically need to cut this man out of your life. If you can't do that, then the question as to whether you really want to remain married is answered for you.