I can think of a couple of genuinely credible risks (as opposed to the 'baby monitor being used by Spectre' nonsense).
- Child wakes alone & is distressed. That's what the monitor's for - just as effective next door as two floors away in a big house.
- Child wakes, climbs out of cot & has accident - once you're out of the room, never mind the house, you aren't really safeguarding against that one.
3.. House fire - yes, horrible nightmarish thought. But approx 300 fatalities (all ages) a year relative to approx 2000 in RTAs, for example. & the vast majority of house fire fatalities have a clearly attributable cause (cigarettes, candles, open fires) & affect middle-aged/elderly adults. Only 21 of these deaths in 2012 were children U16, for example. It really is vanishingly unlikely that your house will catch fire & kill your toddler whilst he's asleep in bed & you're briefly next door, unless you're daft enough to leave a chip pan heating up, smouldering fag butts or lit candles unattended.
4.. Child might be harmed by an evilly-intentioned person - as SGB says, that's far more likely to be the babysitter/your family member/your partner, sadly, than an opportunistic intruder who has somehow deduced that you're not within the bounds of the property. Plus, it's silly magical thinking to assume they'll strike if you're next door with an eggnog but wouldn't dream of it if you're three floors away in a big house.
I don't think that this means that we should all fling ourselves into nightly partying whilst leaving teeny children to fend for themselves, but equally I don't think that it's fair to chuck around accusations of irresponsibility, when statistically, being left to sleep Home Alone with mum carousing with the neighbours, is a much lower risk activity for a small child than getting into his parents' car to be driven to nursery.