Elgin bought the marbles and had a permit to remove them. Before anyone says well it was the Ottomans who sold them to Elgin, not the Greeks, they were not his to sell; you are incorrect. The Attic peninsula had been under Ottoman rule between 1458-1821(or 1830). That is as long as Scotland has been part of the UK. The idea of Greek people being one was yet to come, and even in antiquity was highly contentious.
The idea of Athens as a symbolic heart of the Greek people is a relatively new idea, considering that in the period of their transportation to Britain, Athens was merely a village or small unimportant town.
Whilst of course, no one would support the removal of part of an ancient monument today, this was not the case in the period.
Some may argue that they ought to be reunited with the rest of the Parthenon frieze, however, this itself is not even atop the Acropolis anymore. It is housed in a museum, just like the Elgin Marbles are.
The wider contemporaneous history of the Parthenon was that of rapid decline. Under the Government of the day, it was being looted for use for building materials. In the 17th century, it had been blown up whilst used as a gunpowder store. So it could be said that Elgin's removal of the sculptures indeed may have saved them from further damage such as acid rains that have effected the remaining stones.
To argue that pieces in museums ought to be sent back to their place of origin is ridiculous. If one piece is sent back that was fairly and legally purchased then everyone will want everything back. Remember; Britain has important pieces of our past in foreign museums too. Whilst returning pieces that were proven to have been stolen such as the painting by Klimt, pieces that were fairly purchased have no grounds to stand on.