When DD was younger, this happened to me several times. Ex was determined to be an "equal" parent and used to make decisions like this with no prior communication or discussion.
Like the OP, I used to get really upset and stressed - focusing on how it would damage DD, she'd be out of routine, upset, not have what she needed etc etc.
On a couple of occasions, I saw a solicitor - even filled in the court paperwork. But never filed it. Because DD loves her dad and he loves her, and I couldn't imagine having to explain, in years to come, to DD why I'd got a court order "because Dad didn't bring you back when we agreed he would". She was safe, happy and with a parent. I may not love him, or even agree with the way he parents, but i chose him to father a DD with and accept that I have to live with the consequence of that. He's an arse, but he's a damn sight better parent than many of the kids in my DDs class has.
OP I know it's upsetting, I know it's unjust - but, if you can, try and consider any action you take from the PoV of your DS when he's an adult. You've had some good advice regarding court orders here; they are restrictive, and adversarial. If you go down that route, your DS will undoubtedly ask at some point why it was put in place.
And, despite some upsets and dramas when she was younger, my DD has survived undamaged. She now flits between households freely - depending on what suits her. She has very different relationship with each of us, but her life is enriched by that.