The most ethical shopping you can do is to buy from "sweat shops" in developing countries.
This is because jobs in "sweat shops" are for very many people the best in town and they travel for miles to gain employment in them. If we all boycotted "sweat shops" and they closed down then the former workers would have much poorer, shorter more miserable lives often begging, in prostitution or agrarian agriculture where the Health & Safety standards are even worse than in the sweat shops.
At least in sweat shops, the workers are gathered in larger & better organised institutions which lend themselves to local and international scrutiny. We often get to hear about "scandalous" working conditions and tragic large-scale events arising from sweat shops but rarely do we hear about the 100s of millions of individual impoverished, miserable, short lives with much higher chances of accident & disease for those denied the chance to work in a sweat shop.
If we all buy from sweat shops then there is a very good chance that some of that money is going to those developing countries and to the workers themselves. Plus we are giving them pride & self-accomplishment for earning an honest living and competing with the best in the world for business.
Also, make sure you demand high standards of quality from the sweat shops: that way the owners will be forced to raise the capability of their work-force through training, effective management; and ultimately paying more money to attract & retain the best workers!
And don't forget: 1 or 2 centuries ago the UK was a sweat shop economy; it took many decades of wealth-creation and progress to get where we are now. Let's help the new "sweat shop economies" to join us by buying their stuff. That's a lot more effective than charity & fair-trade!