People are stupid, they don't think before opening their mouths. I have heard that sort of thing said to others and I wonder how anyone can be so personal!
My mother used to say people were selfish if they chose to have no children - and there was nobody more selfish or self centred than her.
The other thing is (not in your case), you don't know if someone is having fertility problems, either primary (or secondary if they have one child); not everybody wants to advertise the fact. In those cases what an absolute pain it must be to have to listen to such comments.
I know several 'only' children who are completely happy and well adjusted, my nephew being one; he has loads of friends, lovely parents (and of course he has us :-).
I don't appear to have quoted anyone here and now can't remember to whom I was replying but never mind, you get the gist.