I think a lot of it stems from either people thinking:
(a) You have more children than you can afford.
(b) You have more children than the planet can afford.
With (a) it is a straightforward question of whether you and your immediate family earn enough or have enough money to support your children. If you rely on benefits, people will feel it is justified to criticise you - they are paying for your children, after all.
With (b) it is more complicated. Maybe you are wealthy and don't need taxpayers' money to raise your kids. But, overpopulation is a serious problem. Every child eats up a little more of the planet's resources. Whether you believe in global warming or not, the planet can only support a finite number of people. It will not kill us as a species in the next 50 years, but it will eventually.
There is also option (c) of course, the perception that the people who have the most children tend to be the least suitable parents. Think of Karen Matthews or Mick Philpott for example. These are the people who spring to the forefront of the mind of many people when they think of those who have had many children.
Not all serial-mothers and serial-fathers are as wicked, evil or downright stupid as those two, but it fuels the belief that people with lots of children are sexually irresponsible, deluded, idiots or money-grabbing scum who will gladly risk their kids' lives and/or wellbeing for the chance of a big cash payout.
The best way you can counter this perception is to demonstrate through your actions that not all serial-parents are like this. You can do this by showing your friend, and society at large, that you can support your own family and that you can raise them to be responsible, well-adjusted adults.
It will take time to change perceptions but you can demonstrate that your friend is wrong only through your actions.