Glad he's back.
As now is an opportune time to talk with him about some of the family issues going on and how he could be happier, I think you should also reflect on your own poor conduct here.
From what I understand from your posts, your son's father was a domestic abuser, and you will go out of your way to avoid speaking to this man, even when your son was missing. Can you see how telling your son to 'piss off to your father's' would be an absolutely awful for him? Words like that spoken in anger won't be forgotten easily or quickly.
I doubt you'd lash out this way to your subsequent children when they are teenagers. So saying this to your son is just telling him that he doesn't really fully belong in his home, in this new blended family. It's quite possible that he's unhappy and acts out with the younger children if he's getting that vibe from either of you.
If this is the case, step one of improving the family situation is for both of you, as the adults here, to really understand his point of view, and for you to not say, or behave, in such a hurtful way.