Frugifanatic... I've been through similar.
Yes the system an abomination. If he gets access, my advice to you is this...keep an eagle eye on your child. Anything he says that might point towards abuse (physical / mental /name calling etc) write it down. Date, time, and what was said. If you notice any bruising... Straight to the doctor... This is very important. Take pictures. Report to SS.
You can then go to the court (I waited all day to see a judge, for an emergency hearing) and ask for a cessation on the court order because you fear your child is being ABUSED. In my case the judge agreed to cease the court order until the SS conducted an investigation.
It was all so long and arduous but my ex seems to have got the message. After being investigated, being forced to attend therapy... He now has contact with our dc again and things seem to be going well.
But I still have my eagle eye. Any change in behaviour, and bruising, any admissions of abuse from my dc... I will be stopping contact again...getting SS involved again...
Basically make it hard for him. Make him see that you will not tolerate the abuse of your son. You will report him again and again, until he either stops, or gets bored and fucks off.