I think a lot would depend on the nature of your children as to whether they'd be happier in a small or large family.
While growing up, I would have loved to have been an only child. Now, I'm very grateful for my sister (brother, not so much) and glad I have her. My mother was an only, and made sure that I wasn't, as she'd hated it so much.
Would I have liked a larger family - God no!
I have 2 DSs. Although every now and then I wish I could have had a little girl, I always swore (as one of 3) that I wouldn't have 3 children - so even if I was younger, no more for me! (Age kind of precludes any more as well)
But I have several friends here with 4+ children - 2 with blended families, but the children all live with the mother - and they all seem to be very happy with having that many children.
I think DS1 would love to have more siblings; not so sure about DS2, he's quite possessive.