My DD aged nearly 12 has been riding out with friends (mostly older) for the past few months. I get her to text me before she leaves and as soon as she gets back, but I find myself checking my phone all the time.
I think once you've made sure they are road- and traffic-aware, and made them promise to always wear hi-viz gear, as long as the pony is good in traffic and they are sensible, you have to let them do it. I have found it very, very stressful, and still do, but the alternative was me walking around the lanes with her (which I did for at least six months before she started hacking out with friends) for ever.
You're not being silly: it is a scary stage. I would never, EVER let her hack out alone, though. (Just in case she falls off/the pony spooks/bolts etc.) Safety in numbers I think. Make sure she does easy routes for a good long while, and gradually builds it up. It's all part of letting them become independent: difficult but necessary.