I honestly think, for the vast majority of people, infidelity is a more visceral pain & sorrow, and more of a fundamental betrayal, than the other shit behaviour you've mentioned. It's sexual/intimate/fundamental/emotional in a way other poor behaviour is not.
I'm not saying other poor behaviour can't cause pain, sorrow, betrayal etc and sink relationships ... Of course it can.
It's just that infidelity seems to be in a league of its own for most people.
You see violent reactions, often resulting in serious injuries and deaths, over infidelity much more often than over any other relationship behaviour.