I would like to reassure posters that you cannot catch old age from clothes.
Toast dresses aren't really my thing (although I love some of their other stuff). But I disagree with the premise that, if you have to ask, you shouldn't do it. I think any departure from your usual style can feel weird and that we let that put us off far too easily.
What I'd say by way of caution is that I think this style of dress requires the best fabric you can afford so that it keeps its shape and doesn't go droopy at the first wash. As far as I'm concerned, Margaret Howell is the maestro at this but that costs squillions and Toast is normally pretty decent.
It also needs to fit well - there's a line between oversized and too big and I haven't fathomed out where it lies but it's definitely there.
I'd ask on the Toast thread too - someone might have bought it and, while they're fans, they don't hold back if they don't like it.