Like bluebubbles, I'm not in agreement with the people who would implicate you in the "destruction" of this marriage, even before any such destruction has occurred.
If this is the sort of person who'd seek out a relationship (or even just sex) outside of marriage, then he'd do it if not with you, then with someone else. It would be his transgression. People who say otherwise are perpetuating a double standard.
What you're consideringa posssible affairis threatening even as an idea to any woman with a partner or a husband. So people here are going to react emotionally, particularly those who worry deep down that their husbands might be the type to have affairs; or those whose marriages were destroyed by an affair. I can understand this.
When affairs are found out, everyone suffers. But (probably) the man suffers least. The wife hates you but not him; and meanwhile he's one who's had a bit of fun on the side. I'm happy, for you, that you've told him you won't consider anything til he's moved out. That's a sensible move from the standpoint of self-protection; but is also shows consideration for his wife.