I always buy new. I had enough hand-me-downs as a child.
I get a lot from E5P and excluding the odd dud I'm happy with the quality. Occasionally it's something from a cheapy shop that I could have bought for cheaper in that shop, but mostly it's an absolute bargain and cost is similar to charity shop prices. I don't feel bad about it being fast fashion, the stuff would have gone to landfill if companies like this didn't purchase it.
I buy the occasional new-with-tags item from charity shops if it catches my eye, I don't go in there specifically to look.
I buy from a couple of catalogues that come in the post because I like the style and it's different to what's in the shops. I also like their basics.
I buy from a couple of niche places online.
I buy from supermarket too and I've been happy with most things.
Like anything, I think it's seeking out what you want. People have an image of certain places that it's only for certain ages and they're too old/young. Or it's "all polyester rubbish" when if you look, there's plenty of 100% cotton or wool blends to be had. Same with patterns, lots of places selling patterned things also sell plain things.
I recently gave a coat to a friend, it still looked like new after ten years use and regular going through the washing machine. It cost £15 from Aldi. There's no need to buy second-hand to get a bargain. If people want to for environmental reasons or because they're looking for a specific brand that's very expensive new, that's up to them, but if that's their motivator perhaps the issues with second-hand buying don't bother them so much?
When I was a teenager and couldn't afford new I often bought charity shop for work and lounging around clothes, but the prices were about half price of even the cheapest shop and there were bargains to be had because they didn't filter off the best to sell on eBay and didn't price according to brand. An afternoon rummaging through the shops was fun then and I'd come across plenty of unknown brands in good condition and colours I liked etc as well as the odd bargain.
I'm unsurprised you find the size issues you do with second hand. Quite often if I'm donating something it's because it's a brand that comes up sized very large or the item has shrunk in the washing machine or tumble dryer. I cut the size label out so people don't think it's their size and buy it without trying it on.