I definitely think you're right to be concerned and careful. A lot of things can happen in that many hours, with people you don't know well. I recall starting my kids off by just letting them go to friends' houses for a couple of hours. Even with that, there were quite a few problems.
One kid's mom went out, and left the neighborhood teenage drug dealer to babysit her kid and mine. Later, she told me about how "hot" she thought he was. Before that, she seemed like a normal adult. I didn't know she was an idiot.
Another time, I went to pick my kid up from another friend's house and, again, no parent was home. (I don't remember what the story was that time). I also didn't like the big, full, glass gun display case in the living room, which wasn't locked. I don't know if any ammunition was in reach but still.
There are a lot of dysfunctional people around. Aside from the nightmare you mention, parents can get drunk or high, fight, leave, or grossly fail to supervise. They can leave things in reach that kids shouldn't have, drugs, weapons, car keys. And sometimes kids like to show off and do naughty things if they get a chance, especially in a group. Sneak out and run around the neighborhood in the middle of the night, etc. There may also be older siblings to worry about.
I did let my kids have sleepovers at my house, though. I saw it as a compromise. I didn't really give reasons why they couldn't spend the night at other people's houses. Just "No, I don't think so. Maybe some other time. Your friend can sleep over here, though." After a while, they just kinda got what the deal was and then they didn't ask very often. I didn't want to make them afraid of everyone or have them possibly repeat to other kids that I might not trust their parents, so I just didn't explain. Not sure if that was right or not but they're in their thirties now and have never even brought it up, if that means anything.
I made sure it was fun when other kids were over. Pizza, snacks, dancing, movies, whatever. Then they grew past that in-between age and it wasn't an issue anymore. I think it's fine to trust your own judgement. Every parent does things a little differently and no kid gets everything they want. :)