That's exactly what I meant.
I'm totally stony broke, but I still manage to do what I can, i.e. buy fairtrade coffee. But I a) there is no Waitrose in NI so I can't shop there and b) I can't afford to buy everything ethically.
I do care about other people. But at this point in my life I have to mostly be concerned about my family and feeding and clothing them.
My son would have no clothes if I was concerned about where every piece of clothing he wears is made. So I shop at Primark and Tesco and ASDA and he's covered and warm and gets to wear the same characters his friends do. He's only 3.10 so not quite worried about fashion yet. 
And that isn't going to change any time soon. I'm one of those hidden disability people the government hates who is too well for DLA (or whatever it is now) and too ill to hold down a job so I freelance when I can get work. My husband makes a fairly good wage but hasn't had a real raise in, probably, 10 years, so not keeping up with the cost of everything. I almost cried when I saw Tesco milk, which I know is ethically produced and fairly paid for, had gone up 2p a litre. That's an additional 12p a week of my grocery money, which sounds like nothing unless you are as tightly budgeted as I am. So I've switched to a brand I don't know the ethos of because both my son and my husband love milk and the other brand is nearly 20p cheaper for 2 litres.
So if you have the time, energy and money to research and be ethical (whatever the fuck that means these days) go for it. I don't and just maybe never will.