I think they're right to flag it up, but it could be a blessing in disguise, and lead to more help.
My twins are 10, and one (t1) was diagnosed with autism and severe learning disabilities, the other (t2) has been on the pathway for years now. T1 is exactly the same wherever he is, which is why he was so easy to diagnosis. He trashes everything and is becoming aggressive.
T1, masks something chronic at school, so it's been harder to get him help. Of course where he's masking, he's saving it up for when he's home (the coke bottle effect), and so there's lots of verbal aggression etc. He is only like that with me, because he feels safe. He knows that, no matter what he does, he'll still be loved, and he won't be physically punished, or punished in an unfair way.
Your son obviously doesn't feel safe enough with his dad and is masking.
Unfortunately this does make it look as if it is your fault, at least to people who don't really understand what this life is like.
My best friends have sons the same age, and they're all ND too. When they go to each other's houses on their own, (i.e. without me) they're as good as gold!
I've had social services involved for pretty much all of my kids' lives. Occasionally due to a malicious report, occasionally due to neighbours not understanding, once due to child protection (my mental health was shocking, and I had no idea how to parent), and now we have a disability social worker for T1. They have been nothing but helpful, and have opened the door to help and advice that we never would have had otherwise.
I'm wondering if your son has a diagnosis of ODD or PDA. Even if not, the PDA website has a lot of good advice for kids like ours, it's called PANDA. And also have a look at the book The Explosive Child by Ross W. Green, it also has a lot of insight and good advice.
And tell the truth about DS' dad, they may want to consider whether he is safe with him.
So please try not to worry, they can see when a parent is trying hard and nothing is working. They might have some ideas that you haven't thought of.