As the eldest of four children I can tell you there are plenty of downsides to having siblings. Yes, if I had been an only I probably would have complained about being lonely and been annoyed with my parents, but children (especially me) always complain about what they don't have, especially if they think they are missing out.
I remember feeling lonely despite having siblings, and it was mainly due to having parents who did not encourage me to mix with other children enough. My closest sibling was four years younger and I saw her as a responsibility or an annoyance rather than a friend - at ages 19 and 23 we are finally approaching something like friendship instead.
The best advice I can give you is to not feel guilty. This is your life, it's your decision whether to burden yourself with multiple children. Other people might criticise but you are the one who will have to live out that reality, not them. In my opinion, it is 'cruel' or 'selfish' to have more children than you can afford, or more children than you will be able to give attention or spend quality time with. Those are the reasons I have decided to only have one.
Also, if your child asks for siblings, tell him/her that you wanted to give all your love and attention to them only. And remember, it's not your child's decision only, it's yours. Make sure they have plenty of playdates with friends and a vibrant social life and they will not truly feel alone in that way.
Additionally, as the eldest of four, I began to feel left out as the years went on and the family grew. The newest baby was always the novelty, and while I still got presents at Christmas etc etc, I feel like my parents began to ignore me more as I grew older. I imagine that if my mother had had less children, we would have had more money to spend on holidays and weekends away (we had absolutely none) and might have been closer as a family. I don't remember feeling jealous at the time, just sad.
I'm having a little boy by C-section on Tuesday and I have vowed to give him all the love and attention he deserves, THROUGHOUT his life and not only while he is small and cute. I can't say for certain that I won't one day give him a sibling, but if I do I personally would want to wait until my first is ten because it would be like having another only, all over again. I don't crave the idea of people carriers and a noisy, chaotic house!