Well, mumsnet users haven't exactly covered themselves in glory here. Let's start by saying the things that shouldn't need saying (But clearly do). Men can be - and are - also victims of domestic abuse. In fact, 1 in 3 victims in the UK is male. Those men are no more to blame for their abuse than female victims. It. Is. Not. Their. Fault. There are a myriad of reasons why people find themselves on the receiving end of abuse, which can come from years of having their self-esteem chipped away at. Male victims are far less likely than female victims to seek help, or tell anybody about the abuse they are experiencing. The reasons for that should be pretty obvious to anybody reading the shameful comments on this thread that lay the blame with the male victim, rather than with his abusers. You are part of the problem.
So, the first part of my advice is aimed at your partner. I would suggest that he starts by calling the Mankind Initiative on 01823 334244. They are a charity that specialises in supporting male victims of abuse, and will be able to help him with both practical and emotional support. He needs to take steps to secure his access to his children - he may be worried about the Family Court, but he should be reassured that a Judge will take a very dim view indeed of his ex's behaviour. Threatening to withhold contact in order to control his behaviour is abusive not just towards him, but also to the children. A court will take that VERY seriously. Beyond that, he needs to take real, practical steps to detach from his family and establish his own life. That means finding a place to live, and finding a new job. None of that will be easy. And so, my final piece of advice to him is something you may not wish to hear, OP. This man should not be in a relationship right now. He has a lot of work to do on himself - emotional work to understand how he has tolerated this, and equip him with healthier boundaries. And practical work, to rebuild his life and secure a positive future for him and his kids. Until he has done that work, he doesn't have the wherewithal for a healthy, positive relationship. In fact, a relatinlonship will only distract him from the work that he needs to do.
So, turning to you. You clearly have a lot of compassion for this man, and care about him deeply. It is possible to do that, while recognising that it is not your job to 'fix' him. He really isn't in a place where he can be a good partner to you right now. Recognising that doesn't mean he isn't a good man, or that - in other circumstances- this couldn't have been a great match. But you need to maintain your own healthy boundaries - and that means recognising that this chap is not ready for a new relationship, and won't be for some time.
Good luck to both of you. And who knows - if this is truly meant to be, then you may find yourselves brought together again in the future. But first, he needs to sort himself out, take control of his life, get the practicalities in place, and do the hard work on himself so that he can come to a new relationship from a healthier place.