I will say, apart from you have my utmost sympathy, that there's the practical to take care of too. If he is not 50/50 you need to understand you will never get any form of child support from him - the 50/50 is often used as a way out of that and he won't pay if you have the child all the time. In his mind why would he pay to lose his child? So there's that to consider, and take into account. And I'm sure you already have.
Then there's how your son will react. You may need to figure out a way to get them apart for an extended period and then when your son is receptive try and explain that things will have to change with contact, and why. It might be a good idea to find a child psychologist if you can afford to - or maybe there's someone through the system? They can help your son and help you understand how to have an age appropriate conversation.
Then you have to go full Monty on it and go no contact, for both of you, except through a solicitor. I'll explain why. As a man he thinks he's 'winning' (fuckwit. I had a business partner like this, total arseholes), and you have to 'negotiate' and the ONLY way is to start from a position of total power, with all the cards. If you give him anything WITHOUT GETTING SOMETHING IN RETURN that's 'weakness'. So you take it all away and you only give back small amounts, contingent on him doing exactly what he should do.
This all takes time to prep and do in the meantime record everything. And try not to worry - yes these people do win in the family courts - but they also lose. Because the courts have seen this behaviour before. They win when you are unhinged and unreasonable and don't have any proof. You are none of these things. He has previous. Get testimony from the child psychologist/other professionals (maybe a statement), check with whoever advises on these things - or post a thread hear or on a law forum.. they are all sitting at home now too! 😁.
My gut feel is you are going to have to figure out a way to reduce contact down, but won't get it much further down than every other weekend, and he will always make your life a misery, and the best outcome would be to get help for your boy and to meet someone new (when you're ready) who will be a non toxic, good influence and a good support for you - plus will make you happy.
He will always be a prick and he will always be your sons partner - don't let it ruin your life, or turn your son into a prick too. If he looks like his dad he will probably take after you personality wise - not scientific but just what I've observed and I'm betting he does look like him (dad wouldn't be so keen otherwise) so that will always win out in the end. Good luck x