Well I don't have any SW qualifications. You don't actually need any to be a parent!
Yes it was a stupid thing to do, and yes OP in my opinion, you are right to be concerned and to express to your DH your concern and insist that it shouldn't happen again.
As I said upthread I did have to leave my nearly 5 yo DD in the house alone once when she was ill and I desperately needed milk and bread, and had absolutely no other way of getting it.
None of us know what kind of preparation your DH put into place in terms of what was said before he left the house, and how he left them.
The important thing here is that your DH comes to understand and accept that it was wrong, that anything could have happened, and it shouldn't be repeated. Perhaps having a "furious" conversation isn't the best way to achieve that.
Is he always this stupid? Or was it a one-off?
Last I heard, there aren't exactly enough SWs to police every single mistake a parent makes. You would need a SW for every family if every potentially harmful incident became a "safeguarding issue".