Shineon, he sounds positively vile.
My ex-P was little better, if at all. I'd left him in charge of DS so I could go to a college interview. As I left, I told him DS needed his milk feed at 1pm, just after lunch, then a nep at half one. I returned at four pm, (I'd been out five and a half hours) to find DS still hadn't eaten lunch, hadn't had his milk, and hadn't had a nappy change or sleep. When asked about it, ex-P told me that DS hadn't "needed" a nappy change, wasn't hungry, forgot about the milk, and he'd spent a whole five minutes trying to get him to take a nap. 
I let the subject drop, giving him the benefit of the doubt (one off, or whatever)
A few weeks later, I was ill, and ex-P said he'd take the reponsibility of getting DS to bed. This was the first time in 13 months he'd tried, so DS wasn't used to it and screamed. I took over, got DS settled, then sat down to watch a film, when DS started crying again, to which ex-P replied "I'll fucking belt him if he doesn't shut the fuck up"... I told him if he ever laid a finger on DS we'd be gone. I settled DS back to sleep, and that was it.
When DS was 16 months old, I'd returned from the hospital after an eye check up. Before I'd left, I'd emptied the ironing pile on to the armchair, to tackle when I got home. DS managed to pull some clothes off the pile in one go, and was greeted with me telling him to stop, and then his Dad grabbing him by the arm. Even if it didn't hurt, it scared him. I still remember the look on his face.
I left the next day, along with DS, and a rucksack of clothes. That was it... I'd forgiven ex-P countless times for assaulting me, but was not gpoing to risk letting him hurt DS. So yeah. Shineon is right. Something can/does kick in when you need to protect your child.