Another 'only with an only' checking in. I enjoyed finding my own tribe in the many activities that I went off to (and which of course my parents could afford more easily with it being just me), and my son (now 14) seems to be doing the same. He enjoys Scouts, and another intensive hobby activitiy which takes him off on camps/ all day holiday workshops. Other families like having him over because he gets on well with everyone (then comes home and enjoys the peace & quiet), and I make sure that our house is welcoming to all his school friends as we obviously have it easier in terms of childcare - I have a mostly term time only job too, which helps. We now also invite one of his friends to join us on holidays so that their siblings can have some undivided parent time. As others have said, our son has the benefit of more of our time/ money than he would have otherwise, so we don't have to think as hard about what we can afford for him.
People do say weird, ignorant and offensive things to parents with only children in the early days, but this fades out after baby/ toddlerhood in my experience.
I believe that one-child families are becoming more usual now, so you can enjoy the greater freedom, flexibility, money, and even the 'saving the planet's resources' aspects of it all.
Finally - I'm sorry you're feeling down about your situation. There's always something to feel guilty about as a parent, but try not to let this be one of them. Get your lovely son signed up to Squirrels (baby Scouts) or kids' drama classes, or children's choir or football - whatever he's into, and he'll get all the friends he needs, then bring him back home to his peaceful house with the undivided love of his mum.