Dippy, I do remember your back-story. I'm not going to repeat it here, it's just awful. I feel very sad that you are still with this man, or at least, still playing happy families in the hopes that it all turns out well. I'm even sadder that his aggressive behaviour (which has been mainfest in every country in which he has lived, in every context) is now coming into your children's lives. This is not surprising, you know what kind of man he is and you need to get him the hell out of yours and your children's lives.
The threat is if you are not there, he won't see the children. Great, this is the best outcome for everyone. Stop saying he's a good dad some of the time, again, I don't intend to list the reasons he's not dad of the year, but he isn't, he just isn't and you do know this deep down.
I don't think people are being quick to judge, more trying to encourage you to continue down the path of divorce and moving him out of your lives, and not to continue to make excuses for him (of course, he's ill and sad now you have called him to account, I'm sure he wasn't when picking up his own child and shaking him). I know you feel sorry for him, I know you loved him once/still do perhaps, but you have to get away for your own sanity and your children's safety. Nothing you have said makes me think this is going to be a one-off, he's moving from the older one to the younger, please protect them.
I suspect unfortunately that your colleagues see the professional you (which is a great side) and are reacting to stereotypes about other professionals which is leading them to miss the abuse and seriousness of the situation you are in. No amount of happy days out will make up for these awful experiences, something like that plants itself indelibly in your mind.
YOu don't need SS to protect your children, this is just a measure of how scared you are that you prefer the authorities to step in and make a child protection plan rather than you set down boundaries with your husband. You are terrified about him finding out that your happy family is actually getting a decree nisi very soon. If I were you, I would call a halt to this fantasy now and ring Women's Aid and get some more concrete advice on how to protect yourself and your children when he finds out what is going to happen.