Is putting a fence up an option? Kids 'round here are a little bit like that but it does work both ways, I do have 5 or 6 kids in the house a lot of the time (3 are mine) but equally the older 2, or one of the older 2, is often at one of a few other houses, all equally casual.
I only allow "drop in" guests water or the fruit that is out (mostly apples and bananas) and my kids also whinge that I don't provide packs of biscuits or sandwiches, or the "nice" fruit like watermelon, strawberries and grapes which they have to ask before taking, but it's tough as it would all go in 30 seconds if I did, and I also won't wait on "drop in" visitors - if the kids have friends around on the spur of the moment, the kids have to be the ones to get them a beaker of water or whatever. It is also "not fair" that when they ask at the last moment, while I'm serving, if child X can stay to tea I say no...
Even though I am mean (and tell the other kids off if I need to too, I would be talking to the parents if they broke something expensive, that doesn't seem to happen) the kids still seem to love coming here, and I do like that the kids have lots of friends - it always sounds a bit suffocating to just have your nuclear family in the house and sad for kids who don't have friends locally to play with, without it being some big hassle and having to be arranged long in advance.
Are the kids who are breaking your kids toys "real friends"? I know one "friend" of DD's says things like "I'll only play with you if you invite me to dinner/ get your mum to give us biscuits" etc. and DD thinks she's cool as she's a bit older - so that "friend" only gets water here and short shrift if she is anything other than lovely, and she can wander off home if she wants anything else! I have also sometimes introduced 1 friend per DC in the house limits from time to time, and also when the weather's nice randomly insist they all stay outdoors, which works fine - I guess you'd have to also keep the toys away though 
A fence sounds an essential addition for you though, unless there are rules preventing that?