They were probably scowling at you because you were scowling at them.
There's more than one supermarket in the world, some have trolley bays that are spaced quite far apart and so would mean a minute or two to return a trolley and perhaps leaving your child out of sight in the car.
Before we moved house our nearest supermarket didn't have a trolley bay within sight of the P&C spaces as those spaces were situated around the corner from the main car park. And the trolleys that are designed to hold a young baby still in a removable car seat and the ones with little infant seats were kept inside the building.
Yes you should make sure that the trolley is secure and not likely to roll away but if you leave your child in the car while you take your trolley back someone somewhere has already judged you OP and decided you have no brains for leaving a child alone in a car even for just 20/30 seconds.
And yes, if I can't pay at the pump I will take my child with me to pay at the petrol station if there is a queue. I can't believe you would leave your child alone in a car next to a machine of flammable liquid in a place where your car could be rear ended quite easily 
.
I do take my sons safety very seriously. I lost my daughter because our car was hit at red traffic lights by a lorry driver who didn't take the time to secure his vehicle. But I still stop at red lights even when my son is in the car. When my son was a year old we were also hit from behind by a driver who had taken his eyes off the road to light a cigarette, as he came around a bend and who admitted to be going faster than the speed limit. His excuse was "I've never seen anybody turn into that road (the entrance to a zoo, so clearly a little used road ) so I wasn't expecting a car to be waiting here."
From those two experiences you might have to forgive me for feeling that my son is safer out of the car, rather than suggest I have no brains to want to keep him with me until I am also ready to get in the car.
People have given you a lot of good reasons why they leave their children sitting in them while they unpack the shopping. If you had hit that trolley it would have been your own fault, you were the driver in the moving vehicle. It's your responsibility to look for things like this and drive accordingly. It's not an excuse to leave a trolley at a dangerous angle, with or without a child sitting in it. But you have to expect and look for trolleys behind cars in car parks, especially in the smaller bays, because it's just not possible to fit them down the side of the cars.