Agree with others who have said that it isn't the extra-marital sex so much as the lying and deceit which is the problem for the primary relationship.
Relationships are complicated and people's feelings about what is acceptable vary massively which is why lying and deceit is destructive because it prevents negotiation of terms which are acceptable and is an imposition of terms unilaterally by one partner.
The destruction of lies and deceit is not limited to infidelity, lies about important aspects of the relationship are always destructive.
Personally I would be less upset by an affair than I would a one night stand which "meant nothing" because I feel I can forgive feelings but not what I would perceive to be a tendency to abuse women.
My own relationship is on poor terms currently because of lots of lying by omission designed to present my unconfident husband as pleasing to me. He is only doing it because he feels I won't want him if he doesn't, him doing it has destroyed my respect, trust and intimacy.
It is lack of honesty, secrecy which affects intimacy and trust not extra-marital sex per se that affects relationships. You could argue that society is not particularly conducive to disclosure of desire and non-harmful management of numerous sexual relationships and that it is possible if society was more tolerant this would naturally reduce the lying and deceit but affairs are still a demonstration of cowardliness and weakness, you can't hand over blame to society,