Op it's really not as simple as one social worker makes a mistake = lose your children.
Not in the slightest suggesting that this would be relevant in your scenario, but a social worker on their own cannot remove children. At most, if they were really worried, they could ask if someone could stay with you or if the kids could stay with family/friend - but if there isn't anyone, they can't make that suggestion as it's really only in scenarios where it's normal/straightforward for the kids (eg they usually stay at grannies every week or so, so if we're worried and want to be on the safe side can they just have an extra sleepover to buy us a bit of time to work out what's going on?)
Any steps that a SW wants to take that is beyond that normal support network requires more justification/more evidence because the impact on the children is greater. So for example, if I wanted the children to stay a night away from a parent in more unusual circs ie like a family member where this wasnt part of their routine, at the very least I'd need your agreement and a senior manager to approve it. If the parent didn't agree I could only do that if the police (specially trained police, not a bobby on the beat) agreed it was necessary which is pretty much risk to life (eg abandoned children, intoxicated parents, very severe neglect). Even that is a maximum 72hrs, and the reason for the timescale is so if it happens on a Friday night SWs can get it before a court on Monday/Tuesday (court isn't available on a weekend). Anything beyond the above requires a court order, for that it's a court hearing where there's one SW/legal rep, reps for parents + reps (guardian) for the child so it's certainly not stacked in the favour of social workers.
I'm not in the slightest suggesting that removing children would be on the mind of a SW picking up the referral you've described. Just trying to dispel the myth that a minority of posters seem to push, they they knew someone who sneezed funny near a social worker and their kids were whipped out under their noses. It doesn't happen.
Also, I can tell you the above and you might still think I'm just biased from being in the system, but as an alternative explanation - we all know councils are skint. Cost of issuing care proceedings (neccesary for removal of a child from parents) - approx £20,000. Cost of a IFA foster placement - anywhere between £800-3000 a month. Cost of a children's home - anywhere from £1500-5000 a WEEK. So if you don't believe it on ethical grounds, please believe it on financial - the state really isn't going to stay involved unless they absolutely have to, because there's an ever increasing number of children who sadly do need a service to keep them safe, and there's not a lot of money in the pot to do so!