very very normal, don't worry. At four many children are still playing alongside others rather than really caring too much about being in groups. Also, the party environment is often hectic and overwhelming, I think it's a very sensible reaction your ds is showing! In a couple of years or even next year you may find he is much keener to be with others and less overawed in the group situation.
My ds was exactly the same at activities, he would rather stick with me than do structured activities with the other kids.
School has totally turned this around for my ds and I'm sure will yours too; when you're not there, like at nursery, he's fine, he'd just rather be with you if you're there, it's got to be a sign that he is well bonded to you and loves the security of your presence. I felt that much more is to be gained by not pushing even one bit, let him be close to you if he wants; it is giving him the security to gain confidence, which he needs to function well when you're not there.