I do not understand this. At. All.
In the OP you say there were a "handful" of people ahead of you. Now it's two. So which is correct?
The man who shouted what he did was well out of order, I mean it's not language to freely use around adults, let alone children, and everyone in the queue who heard him will have formed their own opinions one way or the other.
However, if you were waiting at the bus stop long enough to have missed two busses that were cancelled, and long enough for the queue to be significant by this time, why on earth didn't you and your husband use that time to plan what you were going to do when the bus arrived, long before the bus actually arrived?
This is all that's wrong in society ATM, people do not converse with their fellow planet dwellers - there was nothing to stop you speaking to the people in front of you to let them know you wanted to board the bus first, or however you wanted to play the game.
Unless, of course, I'm the only one who talks to strangers in such situations, in which case UANBU.
To clarify - the man was exceptionally rude to say what he did, but was right to feel annoyed if you had boarded ahead of him, and you need to seriously chill-out now, forget his rudeness, but don't forget what happened next time you find yourself in the same situation.