I wonder about this a lot. Ds, who dp and I adore, has just turned two. I'm one of three; dp (and his mum) is an only.
My brother lives in Australia, and my younger sister has Down syndrome. So, like another poster said, although I have siblings, when my parents need support/care in old age, it will fall to me - and I'll have my sister to look out for too. So the siblings-to-share-responsibility idea can be a myth.
Also, because I've been like a second mum to my sister since she came along when I was five, sometimes - maybe selfishly - I feel I'd like a break from caring some time in my life, and maybe the best way to achieve that is through having just ds.
But, having grown up hearing only child stigma said about other kids - and even about dp in adulthood - I've had doubts about not having another child. I had a nosey online and found this article, which I find pretty reassuring - and keep looking at every now and then! It hasn't completely convinced me - I think I just have an "urge" to have another - but in my mind, I wonder if having just one child is best for our circumstances.
Oh, and dp says he wasn't lonely as a kid. His mum's never been precious with him, and went out of her way to make sure he had plenty of other kids to play with. We, on the other hand, were a Forces family and, despite being one of three, with all the moving around, I did feel lonely sometimes.
Ultimately, it seems to be so little about how many kids you have, and so much about the kind of parents you are.