We had DS and fairly quickly thought we were done. He wasn't even a difficult baby, but it was still hard and we couldn't imagine doing it again with a toddler as well. Then he got to about one, life was getting easier, and we thought maybe we'd been daft, so tried for a second.
The second was ectopic and it confirmed our underlying feelings. I was on a ward with women desperately worried about losing their chance of a baby. I just wanted the surgery done so I could get home to DS.
He's now 13 and I have no regrets. Occasionally I think it would be good for him to have a sibling to play with, but we just encourage him to meet up with friends which probably works better as they have the same interests. And because he's grown up with two very dull parents, his interests match ours - we can take him on holiday to various sightseeing places and he genuinely enjoys it, then joins us for dinner and is able to join in an adult conversation (almost too much so - I got a long explanation of why the Taiwan visit was a bad idea in his opinion!).
I can't imagine throwing another child into the mix. I look at DB who has three daughters, and the age range just means they'll never get to experience what DS has done, because there's always a child that's the wrong age, plus the cost of so many more. DS has been on safari, to the US, to London theatres and restaurants - we'd never have done that with younger children in tow.
And yes, he'll be on his own as he's older etc. But DH's sister has moved to the US so frankly she's no help anyway. DH relies much more on me than he does his sister - hopefully DS will have great relationships when he's older and build his own support network.
Sorry, that was a bit of an essay. TLDR - You're not being selfish, it's not bad for the child either!