He's not despairing, he's angry, abusive and violent towards you.
He didn't love you or he wouldn't have been angry, abusive and violent towards you.
You aren't driving a dagger into his heart. He's angry, abusive and violent towards you.
You aren't 'causing us to get further apart'. He's angry, abusive and violent towards you.
You aren't hurting him. He's angry, abusive and violent towards you.
He's just pissed off that you aren't letting him be angry, abusive and violent towards you. You're messing with his belief that he is perfectly entitled to be angry, abusive and violent towards you. That he has the right to hit you, to smash his way into your house to get to you to be more violent and abusive towards you. That you exist on this planet to serve his desire to have a living being to push, slap, punch, scream, shout and batter. Because it's more satisfying to scare and terrorise and criticise and berate and torture a living, breathing, caring person.
For argument's sake, how about if you thought about what it would be like if instead of you he'd been abusing, he'd done all that to one of your dogs? That the way he hit you was how he treated your beloved animals? That he tried to smash his way into the house so he could batter that same dog again and again? Because he enjoyed the sounds of the dog screaming in pain and fear, the feel and satisfaction of his hand knocking a beautiful, soft, warm, furry muzzle or a set of slender and fragile dog ribs? And that the only reason he wouldn't do it would be because dogs have teeth?
I'm being brutal because, in my opinion, your rather flowery language about distance and his hurting is you romanticising an angry, violent and abusive man whose only reason for not hurting your dogs is that it's more pleasurable to do it to a larger mammal. And there's possibly a bit of sexual thrill for him to do it to the same species as him.