I agree with you.
It's nobody else's business how many children you have, or why.
Before we had our DS we lost two other babies, one to stillbirth and one to prematurity. Now we have DS, who is 9.
Strangers would ask "is he your only one?" and I'd be left deciding how to answer. Saying yes would sometimes lead to "you should have another" and some comment about why. Anything from "it's not fair to leave him on his own" to "because you might have a girl and complete the set."
But worse were the friends and family and acquaintances who know we've lost those babies, and who make the same comments.
You can't leave him on his own. Only children grow up weird. He needs someone to play with. Two makes life easier. Two is more fun. It's not a real family with just the one.
Although some went the other way and said things like "don't you think you should stop now you've got one" or "you're not going to put your Mum through this again now are you?"
Nobody left us alone. People were asking when we were trying again three days after our first child was stillborn. I almost died myself when our premature daughter was born. I was ill for months. It meant I needed operations when pregnant, I still don't think I've physically recovered from that time.
We've had a hard time deciding that DS would be our 'only' child but neither me nor DH could face another pregnancy with all the risks it carries, and this time we had DS to think of as well. I worked so hard to have him, I wasn't going to risk leaving him without me or risk missing him grow up.
How I feel at the time depends on how I answer these people, and as DS has gotten older (and so have I) people have stopped asking so often.