When we've been out walking recently in our fairly small village, my 22 month old has been wandering into front gardens and having a look around. Especially if they have Christmas decorations up, like reindeer or a Santa.
I say to him not to touch and that it's not our stuff, but beyond carrying him away or putting him in his trike (I'm 8 months pregnant so can't carry him if he refuses to walk so we always bring his trike with us, but I give him the option to walk) there's not much I can do to stop him and I'm not particularly inclined to cause a screaming meltdown over a quick little detour that I personally wouldn't be bothered by. He isn't causing any damage.
YABU - You shouldn't let him go into front gardens at all
YANBU - As long as he doesn't break anything or stay too long it's fine