I honestly don't understand this.
The court order was for what - for him to see his ds?
He chooses unilaterally not to see his ds four times in a row. Ds is made available each time, but having given no notice or communication, ex doesn't turn up. Ds is young. He gets used to not seeing his father. He is happy not to see his father, as he doesn't want to.
Then, months (?) later, with minimal notice, ex decides he will turn up this time after all. There is no opportunity and insufficient notice to prepare ds for this change. Surely a court order has to have some balance to it? If either of you break the arrangement, without enormously good reason and certainly not time after time, surely it goes back to neutral and you start again with reaching an agreement on how to proceed bearing in mind the child first and foremost?
And one thing I would mention, which is horrible but I'll say it anyway. I'm sorry. It is the constant threats from ex. The sw has just said again that he was extremely threatening towards you. I think you need to raise the question - to sw, to the senior person who has overall responsibility for safeguarding in your area, to the ombudsman, to anyone who is a professional with any interest in this case: given ex's inability to recognise and meet the needs of ds as noted by ss; given the fact that he doesn't appear to care about ds, as evidenced by his refusal to meet with ds at all unless he has full control of the situation and is taking ds to his own place, alone, away from anyone else who has a duty of care towards ds; given the constant threats against you and the anger and aggression with which he freely and repeatedly expresses them even to a professional - given all the above, can they give you full assurance that they have fully risk-assessed the situation, especially in the light of the things ds has said about what his father has been doing to him, and that they have taken into account the documented occasions in the past where an angry and aggressive ex has harmed their child to get revenge on their ex-partner? Can they promise that there is not even the slightest possibility of ex using ds to hurt you? Can they confidently, in their professional opinion, guarantee that there is no possibility of ex harming ds? If they can, on what do they base this assurance? And are they willing to take full, personal responsibility for each occasion of unsupervised contact?
Supervised contact sounds such a good idea, wouldn't you think!