There are so many different varieties of any fabric, @cornflakebrownie - linen doesn’t have to be a grand wedding guest dress, cotton doesn’t have to be a white blouse.
I wouldn’t usually iron the cotton jersey t shirts, sweatshirts, jogging pants I wear at home, though I do iron corduroy things. For shirts and trousers I try to avoid thin flat poplin and look for thicker cotton twill, or cotton canvas or cotton mixed with silk or linen. All of these are much better ironed - but they’re beautiful to wear and generally look pretty good after a day’s wear. The best things in my wardrobe actually improve with age.
With linen I also look for heavier fabrics. In a previous century I remember using spray starch for my City clothes - I’m less scrupulous now as I no longer need to instil confidence in clients.
I prefer crunchy Shetland wool to cashmere and find it extremely easy to look after. Air jumpers after wearing, iron before wearing, hand wash as rarely as possible. Nothing bobbles or shrinks. (I haven’t shrunk a jumper since the early 90s!)
Because I try to avoid dry cleaners as much as possible I’m wary of unmixed silk, because I know that even with the gentlest washing it will degrade more rapidly than I’d wish - a handful of years rather than decades. But there’s really nothing like a weighty, richly coloured silk shirt for a touch of grandeur.
In all honesty I’ve found synthetic fabrics much more troublesome to maintain, and almost invariably disappointing to wear. (Unless sportswear, obviously.)