There is probably a wasp nearby so the tree is probably not the problem. What's underneath the tree, some wasps nests are low down and even on the ground. But they are attracted to the smell. I have a plum tree in fruit and they are all over the fallen plums
Bloody hate wasps, they should all be smushed IMHO!