We have one and he's lovely. I hate, by the way, the term 'only' or 'onlies'. What a way to think about a child / a person. I also hate the word 'just' (i.e. 'just the one'?)
My biggest pet hate is people who have asked us 'whether we're planning to have a family' (I mean after our son was born, assuming that a 'family' means 2+ children). 
Still unclear whether we'll have another.
For many people has to do with money, personal circumstances including career development, and of course fertility issues.
In our circle there's a number of families with one child, for various reasons. I certainly refrain from asking the reason unless they offer the information themselves as I've been asked more times than I wish to remember 'just the one'???
In the current economic climate and particularly for those living in London with insane house prices & childcare costs, sometimes it's simply very hard to have more than one child. More & more families choose to have none (and yes they're still bloody well families even without children!)
So yes YANBU, I suspect you're right and there are some considerable pluses to having one child which people tend to underestimate spouting out silly cliches about 'sharing', 'selfishness', 'loneliness' etc.