I was contentedly child-free till the age of 40 when I had a child, so I have spent far more of my life childfree and not planning to have a child than I have as a parent, and still feel 'on the side' of the childfree. I had decades of all the usual patronising/insulting stuff about how much I would be missing out on, selfishness, did I want to be a lonely careerwoman zooming about in my flash car, and die alone eaten by my cats? etc etc.
I did not in fact 'change my mind' or 'see the light' -- having a child was a deeply weird, out of character thing for me to have done, and I was perfectly aware of that when we conceived. I am not someone for whom being a parent is a natural state of affairs. I adore my son, find parenthood far more interesting than it looks from the outside, and I have no regrets about the decision I made, but I am absolutely aware that I could have had an equally happy, differently interesting life had I not had him.
Interestingly, you would have thought that the people who bemoaned my selfish childfree state would have been delighted when I had a child and entered the fold of what they saw as 'normality'. But no, the comments moved onto baffled incomprehension that I had no plans to have more than one child. How selfish, lonely onlies, no really a family at all etc etc.
My conclusion is that a lot of people are terrified of difference, and that your decision to do something they consider unusual constitutes a threat to the status quo. Especially if it undermines their sense of 'Oh, everyone does X', and especially if your decision is perceived by them to make your life easier. It's psychologically easier for them if everyone has 2.4 children and does the same joky eyerolling about teething and tantrums and teenagers etc, because That's How Life is Supposed To Be.
They can cope with other people's infertility, because it makes them feel that parenthood is the desired state, but your decision not to become a parent reminds them that parenthood, in the first world with access to contraception and abortion, is not compulsory, and that sometimes the alternative looks awfully attractive, hence the need to dismiss it as selfish/empty etc.