Ignore your family and friends. It's just inverted snobbery and jealously. We went through the same thing with our son, in fact, my sister hardly talks to me anymore, nor do several of the other mothers from my son's primary school!
It's an individual decision for you and your son. Every person is different, every school is different. You really CAN'T generalise. You'll get all kinds of anecdotal "evidence" from people, some saying they were happy at grammar, some saying they weren't. Grammars can be very different. Some are snobbish and high-brow, others are friendly and nurturing. You'll get a feeling during the open days, meetings before the 11+, and the day of the 11+ itself.
We were worried because the local grammar is ancient and austere - dark rooms, dark wood panelled walls, looks more like it's out of a Dicken's novel than a modern school, so initial reactions were a real put-off. Before we went, we looked at the website, and were worried because it gave the impression of being stuffy and too intellectual, i.e. lots of emphasis on Latin and Greek, etc. So we went to the open day thinking it was probably not for our son as he's very "normal", loves playing footie with local kids in the nearby field, loves playing computer games, etc. But what a revelation compared with the other local schools we visited! The teachers and support staff were incredibly friendly, the head of maths happily played a maths board game with my son for half an hour, plenty of emphasis on transition and nurturing, the head teacher himself was walking around and happily talking to the children, guided tours of the school were done by teachers and sixth formers, all of whom really engaged with us. After that, it was a done deal and our son really wanted to go there! Compare that with the comps we visited, where we either wondered around without any guides, with just the odd first year wondering around to help with directions, where the head appeared only for the half hour presentation in the hall, where the staff were in huddles talking amongst themselves and even looking as if they were trying to avoid talking to parents and children.
On the day of the 11+, there were loads of staff around, all trying their best to put the children at ease. They got a drink and biscuit and the teachers supervising told them jokes before they started and again during the breaks between tests. My son told me he actually enjoyed the 11+ day! It was then even more of a done deal!
He's now in his third year and has never been happier, never complains of any bullying or other problems, doing very well and exceeding all our expectations. Got a lovely group of new friends - all perfectly normal, and far less snobby than the primary school kids and parents.
If it's right for you and your son, go for it. Ignore your family and friends. it's your son's education that's at stake.