I am an only child. I had a wonderful childhood - I had lots of friends to stay/visit, and lots of extended family too. I had company (lots of it) whenever I wanted it, but also peace and quiet too. My parents also made sure that I was able to go (eg they would take me and pick me up) to various activities too, where there would be lots of other children eg dancing classes, gymnastics, music lessons, Brownies etc.
I certainly wasn't spoilt in terms of material things, but I didn't go without. There was lots that I would have liked in an ideal world, but I had nice clothes and lovely toys and things. We had a static caravan (those were the days!) and so we didn't go on foreign holidays, but we spent lots of weekends and long school holidays staying in our caravan at the seaside. It was great.
I would love to replicate my childhood for me DS. He is nearly 4, and I have absolutely loves spending time with him, and having one-to-one time with him (I work part-time). But my DS is one of three children, and had an equally happy childhood, and he has always wanted more than one child. Ideally, he would want three. Ideally, I would want one.
So we have compromised - I am 10 weeks pg with our second, who is very much planned. Its taken me ages (DS is nearly 4) to be willing to have another baby. DS is an absolute joy, and I just love playing with him and taking him places, and having so much quality time with him. I really treasure and enjoy it, and can't imagine anything better if I'm honest.
But I don't feel that I can deny DS another child either. We are financially stable, have a large home and there is nothing actually stopping us having two (or more, for that matter) children. I know that when the baby arrives, it will be fantastic, and as I am an only child, and DH's siblings are not showing any signs of being anywhere near ready to have children, it is a decision that we have made.
I know exactly where you're coming from OP.