I think the flashy house/cars reason is plain silly.
If you think he'll struggle to get in/keep up, I see the point, it might upset him.
But the main thing is, why would you want him there?
Surely that is the most important factor, and you don't mention it
. I don't know what 'enrichment' means (and I'm an academic with several years of interviewing for Oxbridge admissions behind me).
Do you mean in terms of things like music, arts, etc? If so, that doesn't require private school though I take the point it might come easier there - or, given that you say he doesn't have a particular talent, it might come better as private tutoring where he could go at his own pace.
Or do you mean extra-curricular activities?
Or is it just snobbery?
Sorry to be so blunt, but it matters. Most private schools still expect some subs for expensive activities, so it's not as if you'd budget only for fees. I am not doing down private schools. I know people who went there for reasons ranging from severe dyslexia to bad bullying, who thrived. But I think you need to be clear what you're getting for your money, and whether or not you could provide the same benefits to your child without making him feel put on the spot with entrance exams.
FWIW, lots of us who don't look 'bright' at 11 or so do very well in later life, and it'd be rotten for him to spend his teens feeling he doesn't measure up.