Those people must be extremely insensitive (including the woman who was an only child) to discuss such an issue, and it seems particularly inappropriate at a work dinner. I am sorry that you had to endure this uncomfortable situation.
There are all kind of reasons for not having more than one child. I lived in the US between 1998 and 2017, and knew many people who had chosen to have one child as the US does not have universal paid maternity leave and many employers do not provide it, and childcare, healthcare, and college education is very expensive as well. You can raise a child in a two-bedroom or even one-bedroom apartment (many people do it in studio apartments in New York), but a second child often requires an expensive move.
I worked in an Ivy League university, and met many academics with as a single child who felt that having more than one would jeopardize their careers, which required working out of office hours and travelling for research and conferences. Many states can fire people "at will", i.e., without compensation or redundancy pay, and financial precarity also contributes. I think that we will see more single-child parents and families in the UK as the cost of living continues to rise and wages have flatlined.
I'd add that recent research (I don't have citations, I'm afraid) has demonstrated that only children are typically happier than their peers who have siblings, and that loneliness is not prevalent. Only children usually get more attention from their parents, are often more confident with adults and in social settings, and often their parents have more money to spend on them. Money is not the most important thing, of course, but it does buy childcare, sports equipment, extra-mural activities, and these things contribute to a child's confidence. I have several close friends who are only children, and they are extremely confident, sociable, and accomplished. Taking care of ageing and sick parents when one is an only child is challenging, but siblings often argue rather than working together, so that's not necessarily ideal, either. I think it's important that parents of only children plan for the future if they are able to do so, but there is no need to feel sadness or guilt about only having one child, whether by choice or circumstance.