I think people are being a bit hard on you OP. It was first thing on a morning and you were met by a surprise. Now you've had time to think about it, you can sit down with your daughter and explain your boundaries.
My kids aged 9 and 6 found the back door keys, went into our back garden, scaled a 6ft gate (so changing your fence to 6ft may not solve your problem) and I found them at 7.30am playing in our front garden. I had no idea that was going on and that fact still scares the crap out of me.
We sat down afterwards and explained to them both how dangerous all that was. But until they've pushed a boundary kids don't necessarily know that something's not ok. And sometimes you, as a parent, can't predict what they're going to do.
As for other people's kids, we're the strict parents locally and have had rumours spread that we're abusive because are kids aren't let allowed out at all hours, so I get the reluctance of people to be seen as strict.
Kids around here are regularly out on bikes in the middle of the road at all.times of day from age 5 years upwards. One neighbour has a child that sings karaoke loudly and out of tune with a large amp any time from 7am to 11pm in their garden ("She's going to be a singer.") A trampoline would be a blissful sound in comparison.
The parents of all these kids tell everyone what great / hippie / gentle / relaxed / understanding parents they are. I did, for a while, copy them because I'm trying to learn how to be a good parent. But then, after nearly knocking a kid off a bike driving down my road when he swerved off the pavement oblivious, I realised it wasn't keeping any of these kids safe and I didn't want my kids to end up hurt. So now, we're the strict "abusive" parents with boundaries. This thread surprises me as we're very much the anomaly with our boundaries.
Ok rant over.
OP, I'd work out what your boundaries are and then sit down to discuss them with your daughter. And then see those boundaries through if the neighbours granddaughter appears over the fence at 7.30am again. Just explain it's too early for your family.