How is it selfish?
Selfishness implies a lack of consideration for others, or concern for your own desires and well-being above those of others.
Well, where are the "others" in this scenario? Who are you being inconsiderate to, op?
I would maybe understand the charge if your partner desperately wanted another child to the point of mental anguish and despair, and you said no. Or your child wanted a brother or sister so badly that they were making themselves physically and psychologically ill over the matter.
But, at the end of the day, it is a woman's right to choose whether or not she wishes to engage in behaviours that may lead to conception, pregnancy, birth and another child, so the former examples are pretty redundant really.
Seems to me as though this fellow mum has opened her mouth and allowed a bit of silliness to fall out.
Next time she says you are selfish, ask her why. And if she says you are only thinking about what you want, ask her who else you should be thinking about when it comes to this matter.