Ah .... the age old excuse for not being open about an affair. If the unknowing, trusting wife finds out, she will be unhappy .... so for her sake she is kept in ignorant "bliss".
Of course it's relevant .... if you are questionning the morality of a situation, you cannot conveniently ignore someone who has a vested interest in the situation .... someone who presumably made vows with a view to abiding by them.
I have no issue with open marriages if both parties consent to that arrangement, but you - and your lover - have no moral right to decide to keep what you're doing from his wife and your husband (I have only skimmed the thread ... maybe your husband does know ?) If both of you feel compelled to seek something outside of what your spouses understand to be the status quo, then that compulsion should be discussed before it's acted upon. Some people might actually be okay with it I suppose, but all evidence suggests that most wouldn't be and that you would therefore need to make a choice about whether to stay or whether to go if your partners found it intolerable .....
.... in which case, by not broaching the issue of being attracted to, (and acting upon that), other people, you are having your cake and eating it - the best of both worlds and so on. You won't be honest, because then there's a (probably high) risk that you will lose out in some way .... which is selfish and is immoral.
I'm certainly no psychologist but the fact your DH has had affairs too probably does contribute to the situation you are in, and your attitude to it, and I do understand that up to a point ..... but what I don't understand is why anyone would deliberately do anything that could potentially devastate someone else's life (do they have kids ?). You have no way of guaranteeing that his wife would never find out so you are gambling here ..... and yes, I know you are not her husband, and that he not you, owes her loyalty etc., but you are enabling him to do the dirty on her.
If you are confident and happy with the way you live your life I genuinely don't understand why you would feel the need to come and philosophise about the morality of affairs on an internet forum. Surely there are only 2 reasons for doing that (if you're not wracked with guilt and therefore seeking a kick up the bum to sort things out) and that is a) you are actually boasting or b) you want the approval of strangers (which would indicate that you do carry some guilt, but want to eat that cake, and are hoping that a bunch of strangers can make you feel better so you can carry on).