The thing is, on the one hand you are calling your child "my baby" and on the other, you're indignant that your ex is calling him half his baby. Only one of you is acknowledging the other parent, and that one isn't you.
The reality is that this is court ordered contact, and if you flout it without good reason, you could end up losing primary residence. Birdsgottafly is wrong in saying any non-resident parent who wants equally shared care will get it, as usually that's awarded where parents co-operate well, and it isn't always going to be in a child's interests. It simply is not the law as it currently stands. But I do think you need to start thinking that the judge and CAFCASS officer here weren't working together to harm your child - they were trying their best to do what was best for him. And right now, that means 2 weekends a fortnight with his father, so he can build a strong relationship and move forward with the security that brings.
Your instinct to protect and nurture your child is human and laudable, and you'd be a worse mother if you didn't feel it. But forgive me... you do need to separate out your anger with your ex from your child's right to a relationship with him. You condemned your child to a travel time when you moved away, and to say you think he should also be deprived of truly high quality time with his father on top... it isn't fair on him IMO. You are with your child almost all the time. Please don't begrudge him the time he needs to form a strong bond with his dad. He left you and that will hurt like hell, but the reality is that he didn't leave his child, and he has proven that by pursuing this through the courts at what I can only imagine is considerable financial and emotional cost. You chose to remove your son from his father's local area, and while you had good reasons, it is pretty cheeky to try to rely on your own choice to separate them geographically when arguing you should also separate them further emotionally. Surely you see that? If the journey isn't fair on your child... why did you create it for him? And if the benefits of the move outweigh that price for him, then I'm afraid that is the price he, and your ex, are going to have to pay. If the travel is indeed unfair on him, then you shouldn't have required it. Please drop that argument because as I just tried to show - it makes you, and not the ex, look bad.
Your child will look to you for his own reaction. If you work on being positive and focus on the huge emotional benefits of a good parent/child bond, then he will settle better into these changes. And all kids with a change of scene play up a little and feel tired. Forgive me, but the temptation for all split couples is to interpret normal childrens' responses to things as meaning a contact arrangement is either bad for them, or unwelcome.
In short, please stop kicking against the status quo, because at this point it's a fait accompli and you have to accept that's what it is. Instead, try to focus on the undeniable benefits, and work with it. I appreciate it isn't what you think is the best for your child but you must see it also holds advantages your preference doesn't, too? So please try to make it work, and avoid your child being ground between two warring parents. That will harm him more than anything else could.