I can understand why you are wary of getting involved with the adults - particularly as they live so close to you.
Difficult in terms of your son, though. If it weren't for the quad bike thing I would think it worth letting him play with the other little boy, but always being cool with the parents so that things didn't go any further with them. But I think the quad bikes pushes it over the edge, so I would probably try to avoid my son making friends with him unless I felt confident that he wouldn't be going on them (so if the other boy only came to play at yours, for example). If not, although I would be sad for the other boy, I would really try to avoid contact.