An interesting question, and I have to say that I think Karma says more or less what I think.
I think open relationships are different, and the same is true for those with mental health / self esteem issues etc...
I also think that everyone can make a mistake once...
But, assuming two healthy, normal adults in a marriage with no other issues, then the one that has the affair, or repeatedly has extra marital sex without his/ her partner's consent does not love their partner in the way that they should. To me, Love is putting the other person first. One of the definitions of love is "to care very much about someone, especially members of your family or close friends". Well, if you are deliberately, and repeatedly doing something that you know will bring the other person harm / upset then how can you be said to be 'caring for another person'? To me, that shows disregard for the partner, and that, for me, is not love.
Freudian Slipper above also makes a good point, its not just the sex which shows the lack of love, but if it is accompanied with lying, then this too shows disregard for the partner's feelings. Again, to me a person who willfully (and again repeatedly) lies to someone for their own benefit (rather than the partner's) then this is not 'caring for them'. It is utter selfishness and it is not love.