To me, love means that you wouldn't want anything bad to happen to them. You care about their happiness. So if you had to do something that they might not like, you'd do it as kindly and carefully as possible. It might hurt them, but you'd try to make it hurt less.
When my exh had his affair, he told his AP that he and I were like brother and sister - that he cared about me, just "not in that way". It's part of the Script. Adulterers know that they should either not sleep with B, or they should leave A and then sleep with B. But the fact is that they are staying with A and sleeping with B. If they were doing that coldheartedly, that would make them a bad person. So their subconscious comes up with a scenario in which they can't leave A because they care for them, but they sleep with B because A's love is not the right sort.
It's also very common for them to backdate this lack of love. My exh decided that when we met, we didn't even fancy each other, and just slept together out of boredom/convenience - no big deal. In reality, he was the first man I slept with, and he knew it was a big deal. He asked me out, and was really excited about it; I remember. But I'm sure he honestly convinced himself that it was not like that.
Even if you are not rewriting history, and you were never as keen on your dh as on the new man, remember that you can't blame that on your dh, so it isn't a good "reason" for holding him on the back burner while you sleep around.