I sort of know what you mean OP.
Toast and also (especially) Plumo have some stuff that you would have to be VERY confident to wear without looking completely deranged. I meant to start a thread about these trousers when I saw them a few months ago but didn't get round to it. Now they're ugly! (/ludicrous/what possessed them) If I wore them I'd look like a baby oompa loompa. I can't imagine who would look good in them or has the guts to wear them.
But, I don't think that's middle-class clothes in general, just one style. With Toast, a lot of the clothes are very sack-like or amish-like but for some women, that's what they want, a lot of older women (not all of course) are done with trying to look sexy and prefer comfort, nice fabrics or an arty look.
I find the Boden "little girl" style less appealing, as it's not really stylish or sexy, just kind of repressed. That said I've had some nice things from Boden, it's not all like that.
Then you have the printed white stuff tunic that can make any woman look like a sack of potatoes – but again they have some better things.
And there are other middle-class looks that aren't like this, like nicely cut suits, silk shirts, classy dresses. Brands like Hobbs and Reiss, Vivienne Westwood, and boutiques like Iris have more of this kind of thing.