Yes, you're right - he is taking away your family, your hopes and dreams.
he is a wanker. 
Please don't make the mistake of thinking that because he may not like himself when he is horrible to you, it is Ok, and he can change.
He can't. It would take years of serious therapy to change a mindset that is as damaged as his.
And that would be in someone who WANTED to change. He sounds like he thinks he can get away with it because each time he does it, you have stayed and this gives him permission, as it were, to up the ante. It also diminishes the respect he has for you (if there is any to begin with).
There are two ways to do this. Stay, and have a wasted half life treading on eggshells with a person you resent and fear in your house, and around your daughter, - or leave, and deal with him as best you can, and move on to the life you deserve - maybe with someone new, eventually, but for now, with your little girl and in a place of safety without this arsehole dragging you both down.