I think you have made a good and very valid point there IsItMeOr.
Also, another point to bear in mind is that years ago people who suffrered secondary infertility would have been far more open to the idea of adopting a baby to "complete" their family. Adopting was far more commonplace then and seen as an ideal solution, before the advent of IVF and other infertility treatment, and also there were far more babies available for adoption as the young girls who keep them nowadays very rarely did years ago.
Children did very little in the way of socialising outside of the home years ago too. Going to a friend's party, or occasionally being invited home to play and have tea after school, or to have a friend to your place, would have been the highspot. Playing out would have been far more common and if you didn't have much in the way of friends around locally to play with then you'd have been stuffed.
Life is far more child-centered and fun for kids these days than it ever was when I was a child.
Nowadays, children always seem to be out doing things - well mine does at least. My DS is either at school mixing with other kids, at various sports clubs and activities evenings and weekends mixing with other kids, or he's playing out with other kids or his cousins. When he's in he's interacting with friends on Facebook/MSN/Skype etc. I think he has more than enough interaction with other kids. We don't always holiday alone and sometimes go with friends and/or family and their kids or if we are on our own he finds new friends when we get there no problem at all. Occasions like Christmas and Easter are always filled with family and friends too.
I can't for the life of me see how my DS misses out on anything by not having a sibling, and nor can he.
Why is it such a big deal and what does make such a huge difference? After all, most people only have one sibling. Why does that one extra child seem to make such a difference to so many people? The answer is (IMHO anyway), that when you boil it down, it doesn't.