When I was pregnant with my last baby, some people were very openly rude. I had a lot of "Oh my God, are you mad? You must need your head seeing to!" type comments, as opposed to "congratulations!"
What on earth are you supposed to say to that? I literally stood their open mouthed, and in the end said something like "Yeh, I know, what are we like?" and then kicked myself on the way home for agreeing with them!
I lost count of the old "Haven't you got a TV?" chestnut!
It's not personal towards parents of one, I don't think. It's a situation that arises when you find yourself face to face with tactless and insensitive people who assume that because they wanted more/less children than you, that you must not know you're own mind!
Just wanted you to know that you're not alone, and to some people, only the number of children that they have themselves is the right number.
The only negative comments I got were from parents of only children, and my own mum 
, although that probably isn't relevant!