I'm pretty sure that Amnesty's support for decriminaliztion stems from wishing to make sex workers safer and less stigmatized. I do think one of the biggest problems involved in making sex work safer and less stigmatized is the insistence that sex work in the form of undertaking paid sexual activity (as opposed to eg selling vibrators in a branch of Ann Summers) is totally unlike any other kind of job.
Other jobs involve facing danger on a daily basis, whether from clients or other hazards such as fire/water/explosives.
Other jobs are damaging, over time, to a worker's physical or mental health.
Other industries are crawling with exploitation and abuse of various kinds.
Other jobs can involve performing tasks which a worker may find distasteful and will only undertake them because s/he is getting paid - no one enjoys wiping up shit, for instance. (Well, OK, maybe there are a few poo fetishists who might consider it a dream job...)
There seems to be this lingering idea that men not only 'need' sex more than women, but that it is vitally important to restrict how much sex they can have. This may stem from a type of capitalist thinking which needs people to be in a constant state of frustration so that they will buy crap they don't actually need at all, or perhaps it has its roots in the superstitous mindset that sex is disgusting and only for procreation - or the mindset behind that one, which is that sexually-frustrated individuals are easy to manipulate in other ways.