Careful, thoughtful sourcing is the responsibility of the consumer, as well as realising that if something is cheap then the production of it was probably damaging to people, animals or both.
Paying more doesn't mean it is ethical though. Loads of retailers have clothes made in the same factories as Primark, the only difference is they charge four times as much for the same goods.
I think yes, you can think about what you buy as a consumer - up to a point, it's a complete minefield. I read this thread and thought "FFS, something else I can't buy." But the primary responsibilty is with the retail businesses to act ethically.