Taking my maternal GPs as an example, who were lower middle class people, I think my Grandmother's mother had nine babies but only 5 or 6 survived to adulthood. My Grandfather's mother had 4 but only three survived. That was normal then. BC was still very hit and miss and hard to obtain. I'm talking around 1900-1925.
My grandparents on the other hand, had only two children and most of their siblings had 3 children or fewer. The same with my paternal GPs. I think this was partly because of WW2, and all the associated lack of opportunity and lack of stability for the five years or so before and after it.
My mother's generation were lucky enough to have the pill, and many women discovered careers outside the home so the birth rate stayed reasonably low in comparison to previous generations.
Very large families (I'm talking 5+) were largely confined to Roman Catholics and/or the lowest socio-economic groups. (and it's hardly surprising they were among the lowest with so many children to feed and clothe, so it was a self-fulfilling prophesy perhaps.) But most people had the typical 2.5 children.
The 'average' family size dipped lower still throughout the 80's and 90's due to the new phenomenon of large numbers of single mums with just one child. They had always existed, obviously, but not in such proliferation.
I think that the some wealthy/super wealthy people are having very large families now because:
a) they are lifestyle status symbol; part of that slightly alternative Boho zeitgeist and the obvious successor/antidote to the Yuppie era. It's what all the rich, cool people do now. They've given up a life of cocaine and champagne in Islington for a life of chicken keeping, vegetable growing, dragging numerous small children with matted hair around Latitude, moving to Norfolk or Gloucestershire, and er...cocaine.
b) if the women themselves have super-duper high powered careers they can afford to employ maternity nurses, live-in nannies, cleaners, private schools or boarding schools etc, and have all the shit taken care of for them, so they just get the fun bits of parenting. Whereas Mr and Mrs Average who do not have any of that stuff at their disposal understandably tire of the drudgery aspect of parenting after 3 kids.
And lastly the birth rate is on the rise again almost exclusively because of recent immigration. Most immigrants over the last ten years come from cultural/religious backgrounds where it is the norm to have large families.
Whereas among the UK population of third generation or more, the birth rate is at an all-time low.