I don't think you are being too protective, even if where you usually walk is safe from cars you need to know that he will listen and stay beside you for when you are not somewhere safe.
Forget other parents if you want him to walk holding the buggy that is perfectly sensible, explain just before going out the door, this is what he WILL do..if not he will have to have reins on his backpack, (have this ready so he knows you are serious)
Follow through regardless, if he doesn't stay holding on to the buggy put the reins on straight away, tell him he can only have them off if he stays walking beside you...it may feel difficult but he will get the message pretty quickly.
Once you have sorted that issue apply the same idea to leaving the playground, tell him in advance, in the morning, that he must come when called and walk with you, or he will have to go to his room once he gets home with none of his toys for 10 minutes/will miss his TV time/ or whatever is most appealing to him.
Be very clear.
If he doeesn't come when you call him, then go and get him firmly by the hand and tell him to hold on to the buggy, let him know that he has ignored you and will have to do the consequence as stated that morning...follow through regardless of any arguing or resistance.
Tell him the same the next morning .....he will probably listen! If not repeat..
At some point all Dcs need to understand they have to do as they are told and have consequences....annd you need to feel he is safe.
Good luck!