I'm one of two, DH one of four, and we always wanted four (which is the number we have been blessed with!). I always felt growing up that our family was a bit too boringly small and neat, and yearned for a bit of chaos. Now I have the chaos I have to remind myself sometimes that it was my choice and that many many people don't have a choice at all about the size of their families.
Both of us were ambivalent about stopping at four but decided to stop, mostly for medical reasons. We did confess to each other the other day that we would have been tempted to go on against medical advice - but steps have already been taken ... We know lots of people with larger families than ours, so we think four is quite restrained, really. Around here the official number of children seems to be three - more middle class than two, but not as expensive or time-consuming as four.
I don't think I will ever stop wishing for another newborn, though. I get teary just thinking about mine when they were so little, and I am famously unsentimental.