The problem is that 'only' (hate that word!) children cannot know what it would have been like to have a sibling, and people who had one or more siblings cannot know what it would have been like to grow up without them. So when it comes to making active decisions about the 'right' number of children we focus too heavily on our own personal experiences growing up.
I have a brother, two years younger, and DH has a sister who is again two years younger. Neither of us have ever gotten on with them particularly well. DH's sister lives very close by and they never speak outside of gatherings at the in-laws' house. Even then, they barely talk to each other. Never had a big falling out or anything, they just don't get on. Nothing in common other than having grown up in the same house - which it seems doesn't necessarily mean a great deal. I had a similar experience myself and have often felt that having a brother hasn't particularly enhanced my life in any way.
We have an 'only'. Most of the time I feel pretty good about that. But I've certainly had a lot of guilt over it, too. Our personal experiences growing up have 'taught' us that DD isn't missing out, because our experiences of having siblings haven't been particularly positive. But if she did have a sibling, maybe she'd be happier for it? Either now, or in the future, or both. Impossible to know for sure.
I think much of it comes down to parenting. My mum didn't have siblings and says she wouldn't wish the experience on her worst enemy (which is a lovely thing to say, given that we've chosen to have one child, but that's my mother...). But she grew up as the obsession of a very narcissistic mother, with a father she adored but who never stood up to her mother and who died relatively young. Her mother, meanwhile, lived to 100 and made Mum's life a misery in many ways until the day she died. Having a sibling would not necessarily have fixed anything, other than perhaps spread the negative attention more thinly. Meanwhile our DD is being raised very differently, and we can only hope that not having any siblings doesn't ever cause her to wish that things had been different. Time will tell!
Growing up with cousins nearby is, I think, a lovely thing for children without siblings. For DD it's almost like all the benefits of a sibling but with none of the downsides - doesn't have to share space, resources, parental attention, etc, but will hopefully always have someone to recount childhood family experiences with and to provide 'family' support in the future. Again no guarantees whatsoever - even with three cousins nearby (plus a fourth at a distance) and one of them very sister-like, they could have an epic falling out at some point and never speak again.
But no single childhood experience will be the same as any other, and I think we all need to stop placing so much weight on family size. There are so many other things that matter far more.