I find people's opinions really interesting on this topic. My DD just started at a state run secondary school after completing primary school at a fee paying school.
Like you, we prioritised education over luxuries to send her there, but decided in the end, as an only child, we needed her to have friends locally for secondary school. Friends who she could walk to school with and see easily at weekends without us taxi-ing her to friends living far and wide, which tends to happen with independant school families.
I wasted so much time over 7 years worrying how we were perceived. Even though I knew it was silly; values matter more in friendships, than financial status. Like many people are suggesting, you (and I did too) are judging the 'rich' for being successful....I never wanted a big career and to climb the ladder, I just wanted to be a Mum. Each to their own.
I will also add that, I know many 'rich' people are in debt up to their eyeballs to keep up appearances....I have a friend who works for a debt charity...they help people set up payment plans when things get out of control...she sees school fees on expenses quite often.
I will say that I feel so much lighter now that I no longer have to pull up in my 11 year old car and stand in the playground, wishing I had made more of an effort with my appearance (but this would have been the same feeling I would have had at state school I imagine).
Only one family in our time there really stood out as judgmental....the one whose daughter has commented twice about our very nice semi detatched houses (we moved half way through our time at the school). The first comment had to be at 5 or 6 years old "why is your house connected to someone elses??" And the other perhaps 3 years later, "why has your house got two numbers?" "One belongs to next doors house, and one to ours", was my reply. The school certainly didn't teach her those thoughts.
That family have asked us to a social event soon, which will be mostly attended by families who still go to independent schools. We haven't seen them for about 2 years. I am in two minds whether to go....I am pretty sure they just want to see how we are managing at state school now....if I have to think this hard about if I want to go or not, it's not worth going, I know! Some of us are just overthinkers.
My daughter had a great start in life, the most beneficial thing I think she got from her primary school is not the education, but confidence, I do feel small class sizes help teachers spend more time on each child, building the bits that need extra help.
My advice: invite the family over and never be ashamed of being you! If they make you feel like you want to spend more time with them, great. If not, you don't need to invite them again.