hello ghosty
I wouldn't say this is an only child thing, though I can imagine it is harder when your child doesn't have an automatic "mate" (who they may hate at home!) to give them a bit of confidence so they can go on the swings together.
I think it's natural, and I as a child was very nervous about playing where other children were playing unless I had one of my sisters with me.
I don't think there is anything wrong and it will just take more time. I don't have the advice that the parent of an only might have, but I really just wanted to reassure I think children with siblings would feel the exact same way except they don't have to.
The only thing I would say (out of experience) is that the "go on, join in" thing can be very counter productive and have a negative effect if you are too insistent. I would try to boost his confidence at home or in other ways. Playing football in the garden with him so that he is confident enough to play with children he doesn't know, for example.
Remember how much everything is exaggerated when you are a child? They have nothing else to think about so when a strange kid does something simple like turns away, or looks at you in what you think is a funny way, it all seems huge and you can absolutely shrink inside, even when you are very young and don't know what is going on.
I think he just sounds sweet and shy and it will come.