That argument might have some traction if it were the legal property of a person who decides to destroy/demolish it and people in the area who loved looking at it as they passed by are distraught, but you may say tough, it's none of their business...
But these men categorically did not own the tree or the wall, and the only right they had was the same as all other members of the public: to enjoy looking at it/photographing it/standing next to it/lightly touching it - but not in any way damaging or destroying it.
Going on that reasoning, if I am an art philistine, does that mean that I can go and slash a Van Gogh painting in a museum - with the 'justification that "it means nothing to me personally"?
Most young children learn from an early age that, if something doesn't belong to you, you don't spoil it. These men who are accused are very, very far from being young children anymore.
Also, if people don't care about the tree, because it's 'just a tree', why should they magically care any more about the wall, which is 'just a load of old stones'?