snowkitten, it's hard. I started making notes of his behaviour, so that I could remind myself that I was not insane (DH also minimised his actions).
You have to be assertive. You must.
I had two moments that broke my heart, when DS1 (11) wrote us a letter asking for it all to stop and then was pleased DH had left, and DS2 (3) bursting into tears when DH was cross with the dog because he was always waiting for DH to be angry and he was so frightened.
It was only then that I realised how far things had gotten. He never hit me, but once he was gone I realised the whole time he was there I changed my behaviour, walked on tenterhooks, tried not to trigger his anger. But his anger could come out at anything, it had nothing to do with me, it was within him.
And I let him talk to me like that by not saying, "Stop it! I will be treated with respect!"
The longer you wait the less energy you will have. You won't ever have the time to gather more, it will be sucked away.
It's like putting money into a fruit machine, the more you put in the less you have, but you still hope the day will come when you win and all your strength will come back. But instead it keeps draining you, and the jackpot never comes.