Some parents are not to be trusted with the safety of their children, whether your husband is one of these only you will know. Has anything happened before? If he is generally pretty safe and this is a one off... then you are most like being unreasonable, if on the other hand he was like mine, not unreasonable. Most parents will make a couple of mistakes and near misses. He was aware enough to shout and prevent an accident at least.
Mine forgot to feed them, forgot to give them drinks regularly, did not think that they needed a nappy changed unless it was pooy, left the baby in the bath to answer the phone, (I heard the phone and ran, knowing he was expecting a call from MIL and he would put that first and found very little toddler standing up in the bath alone) left them in their coat in the car when the sun came out while he was so hot that he took his own coat and jumper off, left them in just a vest in the middle of winter (more than once) while he claimed it was not that cold (in a jumper and under the duvet) thought it was ok to have the big windows open in the second floor while the children were in the room... he had previously let one play with the poo in their nappy and spread it about while he had been in the room so not so hot on supervising them, he shouted at me that he could not look for cars while he was playing with ds when ds nearly got run over... (thankfully I was supervising too and stopped ds) he left things in front of the changing mat risking falls while holding a baby, left the door open and ds fell down the stairs and smacked the 18month old when baby weed on the changing mat.
Oh and social services say it is the responsibility of the non risky parent to protect the children...
Oh and those going on about letting him have the baby from day one.. he did not get to have the baby very much from about day five when he tried to shake the baby and was yelled at from across the room before he could actually do it.
And I had no idea he would be like this with the children before we had them, though for the second I knew I would have to be 100% responsible for their care and safety.