I know exactly how you feel.
Our area is a mixed bag but mostly very affluent. We're very not!
Most of our kids' friends are 'big money' and live in huge houses. I don't want what they have - I like my wee house - but I found myself dreading anyone coming because I was embarrassed for my children even though I knew it didn't really matter.
When our daughter left primary I found myself thinking "Please meet "normal" friends who live in "normal" houses 
I knew she hadn't got my mental memo when I met one of her new school friends for the first and, talking about the big freeze of 2010, she said, "Oh yes, our lake froze that year." Lake? LAKE?!! Who has a lake?? 
Turns out not just lake but substantial forest, acres and acres and a MASSIVE house.
I did wonder what she'd made of our tiny house and postage stamp garden when I dropped her off and realised where she lived but I don't worry now. She loves coming to ours - it's about the kids' friendships not property that matters so I try not to let it bother me and do what I can to get the kids together with friends.
Incidentally, when I met DDs other new friend, her house made Lake Forest Friend's house look like a tiny bungalow. They have staff.
Daughter not a bit bothered and just laughs at the fact she still gets lost going to the loo at her mate's and ribs her about having a butler.
I absolutely get where you're coming from - feel it myself - but people mostly don't think anything of it and the ones who perhaps do are really not worth bothering about.
I keep our shoe box clean and tidy (ish
) and decorate it making the most of what we have - with emphasis on charm rather than grand. 
I take a deep breath and keep telling myself No one cares, no one cares, it's only bricks and mortar...
Don't isolate yourself or your kids over the head of it. Be brave.