This is really difficult.
My partner is a solicitor and deals with a lot of family cases. Social or community services are much maligned but do try and do the very best for everyone involved.
They also operate on the basis of precedent, so tend to adopt the same approach going forward. They evaluate a case, follow a set procedure and make a judgement, but that judgement is heavily influenced by 'experience' and that experience is one they created themselves.
It is futile going to court to fight social services, they will not change their position in court and if they are following a precedent, it's likely the judge will follow suit.
Best bet is to challenge social services through their own channels. Ask them to justify their decision, start a formal complaint if they are unwilling to interact and wherever you find a piece of evidence, which is most likely going to be mere opinion, find a way to challenge that.
Then, you go back to court. If you've got the money and energy.
Easier step to take in the first instance is to ask for supervised visits, or limited access.
Oh and get a good solicitor, one that will work your case not one that just turns up in court and repeats everything you've said to them verbatim.