We know affairs cost so much trouble and upset etc but realistically are we programmed to only be attracted to our partner.
Actually, I'd say the truth is both. We're biologically programmed to be attracted to others. We're also biologically programmed to get jealous. Nature has no problem with double-standards, if that's what gets the genes passed on.
At a purely animal level, we want to mate with others to increase offspring with a variety of genes. But what people arguing this means humans aren't "naturally" monogamous miss, is that jealousy also helps us by ensuring our mate makes the offspring with our genes a priority as opposed to any offspring they have with someone else's genes. Thus, nature "prefers" a route somewhere in between. Unfortunately, this tends to breed conflict. But, again, so long as the genes get passed on, Nature doesn't really give a shit.
People expect a "natural" fair solution, but there isn't one because nature, quite obviously, isn't fair - at least not from a humano-centric point of view.
So, it strikes me, that if we're going to be civilised whilst dealing with two competing natural urges, then we're going to have to take some form of responsibility and repress one or the other for the greater good. We either repress/distract our "mate with many" instinct and practice monogamy. Or we repress our jealous instinct and practice polyamory.
Also, its worth remembering we're not programmed to go to defecate in flush toilets - we have to toilet train toddlers, after all.