I'd generally go for the smaller, as they almost always stretch after wearing, but rarely shrink a whole size after washing.
I went the other way with the trousers I'm 'wearing' right now because they didn't have any smaller ones, and it was a mistake - after a day or two wear they fall off if I walk too quickly!