I am all for “dressing for your lifestyle” and so a fleece or borg jacket definitely is the thing to wear for muddy walks, school runs, etc.
However, my rule is that if I am going to dress in a fleece, etc is to stick to plain colours and styles that indicate I am actually dressing for practicality/outdoors rather than fashion.
The other rule I try to stick to is not to mix practical clothes with fashion clothes - e.g. a dress with a fleece jacket.