I have a 4.5yo here who was a horrendous hitter and he will still lash out physically at times if he loses it, rarely now thankfully (Though I think I will be eternally on the lookout/nervous because of prolonged hitting phase).
Anyway, I personally would ditch the star charts and the punishments (toys and aggression aren't linked and I prefer a logical consequence). At home - help him before it happens, I you and X are playing together I expect you to be kind to her, if you need me I am here to help (you may actually need to stay physically present depending on how bad it is and be ready to 'block' him and give him other options-ask her for the toy, tell her you're playing with it etc). Once he's got these, again, remind him of your expectations before he plays. If he hits/scratches, then remove him to another room or remove. So, aim is stop or prevent him hurt in others rather than punishing him. Emphasise the other person's hurt or happiness depending on behaviour and always see to his hurt sibling first. Connect with him-'looks like you're finding it hard to be gentle today, come in here and help me with dinner'. I know some people will think this is soft, I promise you I am firm and no nonsense about it and make it clear what behaviour is expected. If he's aggressive outside, he comes in.
Let's be clear- a 4yo NT child knows this behaviour is wrong, so my tack (which works with my child who only ups the ante if I get mad) is to be very clear that it's not ok but empathise with the feeling. Get him onside, expect the best.
What do they do in nursery, if he hits/scratches, then he needs to be/sit with worker if he can't be trusted to play with his friends. He'll see the benefits to playing gently and using words/asking for help if mad.
Ahaparenting and Janetlansbury worth a look if you think this approach might suit him. I know some will think I'm mad but tbh he can't ve the happiest if he's behaving like this so being down on him or generally emphasising behaviour over himself as a person are not the way to go. Imho.
Finally I totally get your mortification but he's not you, you're not allowing this and he's four, he'll get there 
