OP, you do understand that you should be able to trust him with her at any given second of the day and night, don't you? And do you also understand that anyone who can't be trusted 100%, 24-7 with your child, shouldn't be trusted with them at all. You say you love her, and I don't doubt that for one second, but you are not doing the best for her by ketting her grow up with a father whose protection is conditional, who will shout at her and potentially risk her life (rough handling can cause disabilities, brain damage and even death. It's a horrifically common occurence). What kind of damsge do you think it will do for her to grow up with the message that her father is only safe to be around, will only be loving to her, when everything is going his way, and he's not troubled in any way? It's pretty much ensuring that she'll be a victim of abuse all her life, from school bullying to domestic abuse as an adult.
Everything nice he does for you or her means absolutely nothing in the face of him handling her roughly and the rest. It wouldn't, for example, restore her sight if he she went blind from rough handling (one of the most common things to happen as a result of rough handling).
One last point - she won't remember his poor treatment of her, but it will affect her. Being treated like that will raise her cortisol levels massively, making her anxious, clingy, 'difficult', etc.