I buy merino thermals - but I use them on mountains and sleeping in tents with Cubs etc. I bought 1 cashmere jumper once (cashmere mix, about 10% I reckon) but DH shrunk it in the wash on its second ever wash

We will spend money on decent gear that will last longer - so proper Helly Hanson sailing jackets for outdoors - because they will actually keep the water out.
But then, I will spend money on getting new zips (I split the zip on my work coat, an Ilse Jacobsen, and got that replaced), repairing tears, replacing buttons (ok, simple tears and buttons I can do myself), and also resoling shoes etc.
My feelings are that I would prefer to get actual good quality things that will do the job and last a long time, than a lot of cheaper ones that need far more replacing.
When DD was small and growing fast, she had LOTS of outfits and lots of those were H&M, Tesco, Dunnes, Penneys/Primark etc - wash and wear things - so we had options if life got in the way of washing and we still needed spares for crèche etc.
Now though, as she's bigger and not growing as fast (although still growing), she is also less hard on going through the knees as well, and we have far fewer outfits and they are of a higher quality. But she does a lot of outdoor activity and sport - hockey, sailing, scouts etc - so they need to do their job to protect her as well as cover her modesty.
DCousins are usually delighted to take cast offs, and I will buy in sales/outlets if I can.
As long as everyone is happy with what they have, then surely that's the important thing.
DCs have enough clothes.
DH has what he is comfortable in and is happy wearing.
You also have what you are comfortable in and will wear fewer things more frequently.
Overall, your budgets are probably similar. As long as, if DH suddenly decides he'd prefer a higher standard of something, he understands that he gets fewer of them, and is happy with that, then it sounds reasonable to me.