Lynley, I have one DS who we have decided will remain a 'singleton'.
I would say go with your gut feeling - if you and your DH feel your family is complete, then it is! Be strong and do not be influenced. It's your life,and your family. People in general can be quite threatened by anything slightly out of the ordinary so trot out the old cliches, but I feel parents who are thoughtful about the impact of being a singleton as you are, are sensitive enough to any difficulties the child may face, thus reducing any difficulties anyway. As you say you have ensured your DS mixes and shares well.
I love being able to focus completely on one child, and know I would feel stressed by having to divide my attention, and wouldn't be able to be the kind of mum I naturally am, eg very attentive and nurturing. Of course mums of more than one can be attentive and nurturing, I'm just saying I would find it so stressful it would make me not such a good mum as I feel I am to my DS. It's about knowing yourself and what makes you best able to give your best parenting.
I have a brother, and we were very close indeed as children, so I know the profound bond siblings can have - and I still think it's OK to miss out on this. It's good, but not so good that I felt I had to give it to my DS, if you see what I mean.
As DS gets older, I do find that as parents our relationship is full-on in every way. We are his daily playmates as well as everything else, so sometimes the play he would be doing with another child focuses around us, which means child and parent don't get so much of a distance between them as they can if child is occupied with two or so siblings!
However I do realise the plus points of being an only - financially I have not had to use childcare for DS, have been able to be a mostly SAHM, can focus on his needs and personality, and in future will be able to let him pursue pretty much any hobbies or interests, no matter how costly or time consuming to get to and from.
sorry such a long post but it's something I have thought LONG and hard about!