There isn't necessarily a problem, unless I have missed something? Some parents will just say, go out and play, and the child will go out and play and if they can play at some nice house with new toys and biscuits then all the better. He will no doubt keep trying if he thinks there is any chance at all of playing round your ds's fun looking garden.
Your ds is not at the age where he plays out, and you clearly parent differently to them, therefore this isn't a workable thing, is it? You know this, you would just rather not have the faff of telling him over and over, so you must Be Quite Clear.
This is tiresome, and it won't be the first time if you live on an estate. No-one plays out round here as the roads are so busy. Older ones go to the park.