I've been having this paranoia too. We don't even own the house, it's rented; we've been there a while (11 years) and the landlord hasn't decorated/replaced threadbare carpets once since we've been there. It's got to the point where we've contacted the letting agent on several occasions to ask if we can get on and decorate it ourselves, but no response to any of our requests. So the place looks run down and shabby. Plus, with a baby on the way we don't really want to shell out almost £1500 on a deposit for a new place, let alone the hassle of moving and settling before he arrives!
Again, we are reasonably well off (both work FT and earn above national average salary) but- like thousands of other decently earning, professional couples- we'll never be able to afford our own house as prices in the area are inflated beyond all reason.
I'm already dreading having other children over for 'play dates' as I'm literally ashamed of our house and the state it's in. As it is I'm not 100% happy with the prospect of a new baby rolling/crawling around on grotty old threadbare carpets but, what can we do? It's not how I'd choose to live if I had my own place, but them's the breaks!
I suppose as long as your child has a loving family and a safe place to sleep at night, that's the bare minimum anyone can expect. There's more to life than 'keeping up with the Joneses'. Maybe I should take my own advice!? 
And, as someone else pointed out, if kids are going to pick on someone, they'll find some reason- if it's not one thing it'll be another!