I have a large back garden, large enough to have a trampoline, swing set with climbing ropes, cargo net and shoot. There is also a skateboard ramp type thing. This part of the garden isn't visible from the side gate.
Over the summer dss13 used the garden with his friends so the side gate wasn't always locked (with padlock and key). DH actually lost the padlock at one point so it was unlocked all the time when we were at work and dss13 was at summer clubs.
As the schools went back last week I asked dss19 to pick up a padlock and lock up before going out Friday morning. Dss19 picked one up and returned home, locked the side gate (without going into the garden). He was wearing his headphones (this is relevant).
I got a call from nice neighbour as he heard shouting at my front door with other neighbour banging on it. He was gone by the time I got home but left a note.
Essentially his DW had let their 3dcs (under 7) play in my garden (without our permission). This is something that she had done many times before when we were out. She then tried to leave but the side gate was locked. She had told her DH that she shouted at DSS who locked her in anyway. She got upset called him very upset as it was raining, kids crying etc.
Her DH had to leave work and get ladder to get them out of garden. The side gate is metal and is part of our garden wall. The note said that we were out of order, we have traumatized the DCs. He intends to report us to the police for unlawful detention. 
Both DH and I have tried to speak with him and his DW over the weekend but they never answered their door. Dss said he didn't hear anything as he had his music on. I have suggested to DH we write them a letter. Telling them they were trespassing and shouldn't have been in the garden and that we are not responsible for what happened. DH also wants to preemptively report the neighbours to the police just encase. I think the neighbours are all talk and won't have went to the police themselves. DH disagrees. Should we report them?