Latebiscuit - here are my initial thoughts on why bankers are paid the amounts they are, although it's still a work in progress and I may add to it/revise it in due course. Way back when I first started working in tax I did in fact do a small amount of remuneration benchmarking, so I'm dredging up old knowledge here.
Very broadly, the value of any macro-economic transaction, including pay negotiations, is determined by supply and demand. If demand is greater than supply then the value (wages) go up; if supply is greater than demand then the value goes down.
That's why footballers are paid a fortune - there are very, very few people in the world with the necessary skills to score goals for Manchester United say. And it's why even quite well known TV actors are paid surprisingly little - loads of people want to be actors and many of them have the skills to recite lines successfully. It's also why the length of training required to perform a job affects how much it's paid - because long training reduces supply of qualified staff.
The demand element is made up of two subsets - demand for someone (anyone) with the right skill set to fill the post, and demand for a specific person to fill the post, perhaps because s/he has skills sufficiently above the norm that they are worth paying a premium. That's why the elite of TV and film can earn megabucks - Jonathan Ross brings skills (and hence viewers) to his chat shows that others just can't.
The wages that a specific job commands also depends on what are sometimes called hygiene factors. Jobs which require unsocial hours or unpleasant physical conditions generally command more than 9-5 jobs in nice warm offices - think of dustbin men, for example.
What doesn't affect wages is how "valuable" a job is to society. Footballers will always be paid more than teachers or nurses.
Turning to bankers specifically.
First, there are very few people with the necessary atributes to be a banker -intellectual capacity, speed of reaction, ability to cope with extremely high stress - and not all of those with the right attributes actually want to be bankers. That means that the supply of suitable people is very low.
Secondly, bankers, and traders in particular, can make fortunes for their employers in a matter of days through speculation or deals. And so demand is very high. (You could argue, of course, that demand shouldn't be so high, that banks should forgoe profits earned through certain behaviours that society deems unacceptably risky or otherwise immoral, but that's an entirely separate argument - one that Rabbitstew raises above.)
Thirdly, the job is incredibly high stress and involves hellishly long hours, which means that hygiene factors also come into play.
Fourthly, and this point is my opinion rather than demostrable fact, I suspect that the "jobs for the boys" culture also inflates wages because it prevents challenge of the status quo, and hence change.
Hope that all makes sense. Happy to be challenged on my logic by anyone interested!