With regards hypothermia as a cause of death for babies, this is feasible.
Babies cannot regulate their own body temperature effectively, so they're more susceptible to external factors and can experience hypothermia more easily than older children or adults. In populations such as the displaced population in Gaza, babies have multiple risk factors including lack of shelter (a fair number of babies are effectively having to sleep rough), lack of appropriate clothing, malnutrition and lower body fat percentages, other health issues, etc.
Environmentally, the low temperatures may on paper look not that bad, but those temperatures are then complicated by wind and water. Again, a lack of shelter means people are being exposed for a prolonged period, and at a time when they're going to be moving less and have a drop in body temperature anyway (overnight and asleep).
Adequate shelter and nutrition are so important. Without them, babies and young children die of hypothermia in conflict and disaster zones (this isn't new or unique to Palestine). I actually find it really sad that people are blaming parents and feeding into the idea of parents from certain ethnic or religious groups being less competent or committed to their children. The majority of parents will be doing all they can to try to protect their babies - but in conflict and disaster zones, infant mortality is high.