I have quite low self-esteem and this means that I often dread social situations where I feel I have to deal with them myself (i.e. without my partner there.)
My DS is only little, but I have got myself really down at times worrying about the fact that we hardly spend any time with other children, purely because I fin d it too difficult.
What I would say to you is that if you feel it is a problem and it is getting you down you need to do something about it. Your DS will make friends of his own accord, he sounds like he is perfectly happy with the situation as it stands. But if it is effecting you then that will start to have an impact on your family life too.
I'm sure you are the same as me in that this becomes a vicious circle, the worse you feel about the less you feel able to do, so the worse you feel about it - that's how you end up being depressed.
Do you think that going to see someone about your esteem issues would help? Toe-curling idea to say the least, I know, but it sounds like this could be a downward spiral for you.
When I feel bad I literally force myself to go out with DS... at the moment we are managing one morning a week at a playgroup. I still feel nervous and a bit anxious when I get there but I balance this out by watching how much fun he is having.
As your DS is older, are there not any clubs or activities he can take part in that don't actually involve you meeting other parents? Surely any sports clubs, or something like scouts or woodcraft folk don't require a huge amount of involvement from you other than dropping off/collecting him?