This takes the biscuit in terms of what I've seen judged on MN! Yes YABU you judgy judgy judgerson.
You really think the parent's primary concern should be the fact that you might be annoyed, not dealing with their child (often you have to try to stop a child hurting themselves during a tantrum, not to mention that trying to move a rigid angry child can result in injury to the parent!)
How would they even know for sure that you were in your garden possibly being disturbed? If they did check I bet you would complain they were spying on you!