If the parents organising the party knew that the OP's son had been left and did not have a parent with them, then they were, as piglet says, in loco parentis. They were responsible for his safety and wellbeing whilst he was at the party, and part of that is ensuring that he is collected by the correct person at the end of the party. If there had been some sort of emergency during the party, it would have been the party parents who were responsible for making sure all the children attending the party were safe.
When I was hosting parties like this, I had a list of all the children I was expecting, and marked them off when they arrived, so I knew who was there. I also made sure I didn't hand over a party bag until the child's parent was with them at the end of the party, that way I knew they were leaving with the right person. There is no way I would have driven off if I knew a child had not been collected.
I do have to wonder, though, whether the OP's dp got the end time of the party wrong - maybe it was half a hour earlier than he thought, so whilst he thought he was only 6 minutes late, in fact he was 36 minutes late - which would explain how all the other children had been collected, and the party parents were leaving when he arrived.
But nothing justifies them driving off leaving a 5 year old, for whom they were responsible, alone in a car park, or even in a sports hall.
The only way they wouldn't be responsible would be if they didn't know that the OP's son was unattended at the party - ie if they thought his dad was there with him. This could happen if the OP's dp hadn't handed the child over properly to one of the party parents.
Perhaps they thought the lad had been collected by a parent - and they might be utterly horrified to realise they'd left an unattended 5 year old to wander a sports hall and its car park.