From what you've said, I'd say the odds are high that your ex will never take this to Court. But, if he does, there is a huge amount of help there for you as a family to ensure that the arrangements the Court orders are safe and supportive. A Family Court Judge's only priority will be the wellbeing of your child - they will always try to maintain a relationship between children and their parents, but they have a range of tools and conditions available to them to ensure that contact is safe (including ordering supervised contact only).
You don't say whether Social Services are still involved with you. I would suggest that you maintain that engagement, because their views will be key. If the case goes to Court, the Judge will request a report from an organisation called CAFCASS - they examine all the facts of the case, and advise the Court on things that might be relevant. That would definitely include any risks that your ex poses to your son. Given the conclusions Social Services have reached on the risk of unsupervised access, that will be a very important factor - so you need to ensure that, if it does go to Court, your CAFCASS case officer is aware of the SS involvement, and talks to your social worker. Even if this doesn't go to Court, the continuing involvement of Social Services and other support arrangements will help to ensure that your son is protected.
If Social Services aren't still involved, then talk to your health visitor about an Early Help Assessment - it's a tool that can be used to identify support that a family needs that falls short of full Social Services monitoring. This was invaluable to me when my ex's mental health issues were causing her to engage in seriously erratic behaviour, and following a couple of assaults on me. The mere fact of the authorities' involvement can be enough to rein in the worst behaviour.
Those things will help to ensure that your son is protected through all of this, and that any Court hearing has the information it needs to make an Order that is safe and appropriate. But the Family Court will not address issues around the involvement that you personally have with your ex. If you feel at all threatened in your interactions with him, please report it to the Police immediately. And you may wish to speak to your lawyer about a Non-Molestation Order. You can find information about those at www.gov.uk/injunction-domestic-violence - it's a way for a Court to impose restrictions on your ex to manage any risk that he poses to you. You might also find it helpful to speak to a domestic violence charity, who can help guide you through ways you can protect yourself, as well as offering the emotional support that you will need.
Finally, it may be impertinent (and it is not what you asked about), but I would suggest you seek out some sort of therapy before you even consider getting into another relationship. You spent a lot of time tolerating the intolerable in this relationship. It is always worth doing the work on yourself to understand why that was, and to address any issues with self confidence etc before you consider getting involved with anybody else, to remove the risk of repeating the cycle. Just to be clear - this isn't blaming you for his behaviour - it's about understanding yourself, and getting yourself to a healthy place so you never again find yourself in that position. Trust me when I say that this is a lot of work, but totally worth it!
Hope that helps a little!