Am I the rude one?

[PonyPals]

Would really love to know if I was the one?

Was at a playground with my DS. There were lots of building blocks for the kids to use. Two kids used all the blocks to build a tower (it took them awhile and they didn't want any other kids help) fair enough my DS played with other things for 20 min waiting for the kids to finish.
The boys ran away and I assumed they went home/else where so I told DS to run and slam into the blocks (foam) so we can start building something again.
Well when that happened I heard a scream and I turned around and the mother of the boys was yelling "how could you do this?" I said I am so sorry I thought the boys have finished and left. She said I should be apologising to her children.
I just turned back and loudly said to my DS to start playing as it's his turn now and we should all share the blocks.
I know it's a first world problem but still bugging me. Was I in the wrong? Should I have not encouraged DS to knock it all down?

Just to clarify, I didn't think it was ok that my son's friend knocked the head off of the snowman -- I thought it was ridiculous that the father felt the need to come talk to me about it. The kids were all playing together (and probably built the snowman together, cant' really remember). When I was growing up, parents were nonexistent and kids worked these things out with each other, not by adults intervening. I posted that to show that I understood how absurd the other mother screaming about the block tower was. However, she was upset, so if it had been me, I would have apologized and then rolled my eyes once my back had been turned. When my neighbor came over to discuss the decapitation, I apologized on behalf of our guest (even though I thought, this is hardly even worth discussing...).

My point is, apologizing is what makes things flow between people. We don't all think the same way about things, but if someone is annoyed, just apologize. I would also have apologized if my DC3 had pushed past the baby while reassuring my DS that it was okay. Haven't we all been stupidly protective of our infants? Apologizing is free and easy. I don't feel diminished by it as some people seem to.

She is mental. I’d have given her a mouthful.