wish I had left at the height of all the trouble
See I think this is a very common course of events and you need to turn it around.
People react differently to extreme situations. Some people simply aren't the type to make huge changes under stress - it would create MORE stress for them. Others are the opposite.
You have moved at your own pace, which is always the right thing to do as everyone needs to have that peace of mind that they knew they were doing the right thing deep down. If you'd left then, you might well have ended up with a constant 'But maybe I should have tried harder' loop running in your head.
You're done. You don't have to justify that - no-one does. Although in this case, if you wanted to you could justify it a million times over:
Serially unfaithful (whether 'only' emotionally, or more)
Aggressive.
Doesn't listen to you, ie doesn't respect you.
Just plain unpleasant!
What's happening here is that you are now being guilted into staying by a man who's spent your entire relationship shitting on you, and is now scared to see some chickens come home to roost. He doesn't want to be on his own and responsible for everything alone. But you don't like him any more and don't want to spend your life with him and it's his own damn fault because he's simply not very nice.
You won't ever be happy, you will always feel like this - or gradually feel worse as you get older especially and naturally spend more time together. If he ends up in worse health than you and you have to give up elements of your life to care for him, oh boy will you hate your life then!
This is what that tiny incident has crystallised for you. Such a small thing - and yet it sums everything up. He's just not nice. If you met him now, you'd think 'What an unpleasant little man' and resolve to not get to know him any better.
You've got one life. Please leave him and make that life a good one!