Well, I suppose you're technically right. You are only hurting one person, rather than breaking up a family and significantly changing the upbringing of some kids (and compounding the pain of a partner who's been betrayed by making them watch the effect on their kids if the relationship breakdown).
But, it's a bit like saying that attacking someone and breaking their leg is less bad that killing them. It is, bit both are completely shitty things to do.