Well, as DS1 said he'd seen his brother touching it and it should be put up out of reach I'd have reinforced that by saying yes, he was right - thank you/well done.
However, at other peoples houses I wouldn't expect them to put things out of reach - it's their house and you are all there supervising them surely?
Beyond that, it really, really does depend on the children. All but one of the kids I know would be fine, but the other one is a right little sod and he can't be left alone in an empty room without causing trouble - I think his parents will still be hiding matches when he's 20!!
We have matches in the kitchen drawer, most of my friends do - so definitely 'not out of reach' and the kids know where they are, but also not just lying around on the coffee table etc where the risk of temptation might be high. However, when we go camping etc the matches are always around and the kids don't bother with them.
So, I think you are being a bit OTT, but they're your kids, you're allowed to be :)