Perhaps the not being invited back is because some of those families live in much smaller, cramped accommodation. Some families don't 'invite people back' - when I was growing up on a council estate, children generally played outside, we didn't spend time in each others' homes, and I think that's still the same today. Amongst my children's friends, generally the more middle class ones invite each other back or go to more organised or paid-for activities, the 'working class' or poor ones hang around on the streets.
The segregation aspect is interesting. I think it probably comes down to who you feel comfortable with. But I still think it's worth it just so you do mix to some degree and are aware of how other people live.
My children's secondary school is totally non-selective, so you get every sort of child and family there is. Children do mix, but I think generally find their friends who have similar interests and often backgrounds to theirs.
In Year 7, my son's friends included a millionaire's son and a boy from one of the poorest, most dysfunctional families going - literally every social problem you could think of was there in that family. Within a short time he was less good friends with both - although still on speaking terms - the millionaire's son because he turned out to be too arrogant and just not very nice, the poor boy because his behaviour deteriorated (largely due to family problems), and he started getting involved in things my son didn't want to be a part of. So now my son's friends are generally those who share the same interests as him and I suppose who come from educated, more professional families. It's not so much an income thing - that varies a lot, and many are much better off than us - but more about attitudes.
I have always believed that parents and home environment are the biggest factor in how well a child will do at school.