I read a good post on here once - basically that you get unhappy families of any size - it's just than if there is 3 children it gets attributed to this.
I have a sibling and we never got on. Luck of the draw, different personalities. We would both have been much happier without each other and barely have any contact now. Though of course there are many people who are close to their siblings.
We were similar to you, initially not bothered, then had one, thought we "should" have two, but then realised we were actually really happy with just one. Pros for us: It's so easy, comfortable, we have plenty of money and time, work is much easier (half the time off for sick children, school events etc!), we can fit things around him rather than trying to please two children. I can work part-time and pick him up from school. We can retire early. Quality of life for all of us is much better.
DS is genuinely happy being an only but even if he wasn't, we wouldn't have a second just for him.
It seems very common at DS's school and among people I know, no-one comments on it. I think the only comment we've ever had was from someone from a different culture where only children are unusual, and even then it was just curiosity. Maybe I don't know many rude people 
In every family I know with elderly / infirm parents who need care, it ends up falling to one child, regardless of the number of children. I already know this will be the case with my own parents despite having a sibling.
Have a second only if you and your partner really want to.