I agree alicelidl but would add asking someone if they want to have more and flying off a "oh you say that now" is also just small talk. Annoying yes, and further than most would go, including me, but not something to take so personally.
I've been asked lots of times by other mums, friends etc if we think we'll have more, and I've always given an honest answer, or "oh well, we'll see" I hear an awful lot of people openly asking it to others as well.
When I had my first a couple of people said I "should have lots more" - I took it as a passing compliment, I wouldn't base my family planning on it though and they probably didn't give it a thought after.
These things get thrown at you all the time, usually you just say something equally inane back and get on with your day. See also:
Are you sure you're not having twins/getting your dates wrong/having a baby elephant
You look huge/neat/glowing/tired
You've got your hands full!
Ahhh, brothers/sisters/one of each, that's nice
Are you hoping for a boy? (already have a girl/girls)
Are you hoping for a girl? (already have a boy/boys)
Just you wait until the baby's here/they start getting teeth/walking/talking/start school/are teenagers
Tum te tum until you no longer have small children and everyone goes back to talking to you about the sodding weather 