This is why, in Jersey (not sure about Guernsey) it's illegal to leave the scene of an accident, or even move your car an inch, if an accident occurs which damages either another car, a property or a person.
Calling the police is mandatory. Taking photos is also encouraged if you don't have a dashcam.
The last time I drove in the UK a few years ago, I had a man drive into me in Devon - I was on a main road, he was exiting from a minor road, headphones in, not looking. I stopped as soon as I could see he wasn't paying attention, but he ran into the front of my car all the same (luckily not badly, thankfully).
But because my instincts were to stop, wait for the police, take photos etc, it was only when I had other people caught up behind us telling me to "get the fuck out of the way and move your car", and remembered it's not like that on the mainland. Over here, you absolutely do not move your car unless or until instructed to do so by the police (either States police or the parish honoraries).
After that I swore never to drive in the UK again without a dashcam, which I got fitted straight away. The other driver did try to argue the toss, and it took a few months, but in the end I persisted and his insurance company paid with no stain on my no claims or premiums.
But the yellow box thing in the OP's case does muddy the waters. I might not live somewhere with many yello boxes (I think we've got 2!) but I do know the rules. You can only enter if your exit is completely clear. It doesn't matter if the traffic in front of you is 'flowing', the exit still needs to be clear.
The OP might be on shaky ground here. 