YANBU. The child AND the mother sound like entitled arseholes.
During the time my daughter (now in her mid 20s) was primary school age, (between 6 and 11 mostly,) DH and I had a load of mums behaving like this. Palming their kids off on us, and not giving a fuck if their kid had a coat, or suitable footwear, or any food, or money.
In addition, they NEVER came to pick them up when they were meant to, and we often had to drop them off, and on some occasions, they weren't even IN, and we ended up keeping said child for an extra 3 or 4 hours, sometimes even overnight.
We fell out (big time) with ONE mother, when she did this to us. Her daughter came at 3pm on a day when me and DH and DD were going out for a meal for our wedding anniversary. DD begged for this friend to be allowed in 'just for an hour...'
She was still at our house at 5.30pm. I said to the girl that she needs to go home now, as we were getting ready to go out soon. We walked her home, (she lived ten minutes walk away,) but no-one was in. So I rang her mum. No answer... SHE rang her mum. No answer!
Over the next hour, we kept ringing, and also popped round to the girl's house two more times. No-one in. House bolted up.
We were fucking fuming. 7pm came and the girl was still here. No answer on the phone OR the door. We went to her nan's house, 3 miles away, (and no-one in there either.)
We were meant to be going out at 7.30-7.45pm, and the girl was still at our house. Our table was booked for 8pm, and we missed it... I was fucking incensed and so was DH.
At 8.30pm, still nothing. No-one in, and no-one answering the phone. DH rang the mum and left a message on her voicemail saying he can only assume she has abandoned her daughter, as no-one is in, and no-one is answering the phone, even at 8.30 at night. So if she doesn't come to get her within the next half hour, he's calling the police.'
Twenty minutes later, the mum turned up. She screamed 'Get here Poppy! Fuxake!' Then she glared at us, called us cunts, and said 'how dare you threaten me with calling the fucking police!' with an angry
glare as she stomped off.
We were so shocked, like WTF ??? She is the disgusting irresponsible 'mother' who doesn't give a fuck about her daughter, yet we were being screamed at by her...'
Sorry, that was a bit of a rant. 
This story just reminded me of this!
Oh, our DD still sees this girl (now woman,) now and again, and she left home at 17, never went back, and has had nothing to do with her 'mother' for ten years. Shocker. 
@Oilpyii PLEASE don't become the same kind of doormat/mug that I was for quite a few years when DD was primary school age. Many parents will take the piss, because they can't be arsed with their OWN children! Glad to see you made the child go home ... Continue to be strong!