The social worker at this stage was just basically asking about family tree, what my wishes are. I explained that I do not want to be in a relationship with him, would like our lives to be separate and one day would want him to have supervised contact with our children.
Remember OP that her priority is ensuing the children are safeguarded.
I would be saying that you want separate lives and are aware that if he were to pursue contact it would need to go through the courts as he is a violent offender and you would want to feel confident that his suitability for contact was robustly vetted and that a supervision plan would be put in place, before considering him having contact with them.
Otherwise if you say 'ideally I'd like him to end up having supervised contact' it can be deemed as you prioritising contact over safeguarding if that makes sense?
I know that's not what you're doing but you need to show from right now and onwards, a consistent commitment to keeping that man away from your children unless the court demands you facilitate contact in which case you'll comply but only after all safeguarding measures are put in place.
She's asked your wishes. I would be saying something like "my priority is the kids and I being safe, whatever that takes. I'm happy to be relocated for our safety and to comply with the system to ensure as many safeguards are in place as possible at every stage of the next months and years. Contact will obviously be something the courts will need to show is safe as I do not want direct contact with him and I am not comfortable with him being around the children unless officials have deemed him safe to have supervised contact and they can walk me through all the safeguarding measures there would be to protect the kids." Etc. And mean it!
Your SW is going to be checking your stance isn't softening / you aren't at risk of allowing him contact that could put you, the kids and the court process in danger.
I hope that all made sense, sorry if it was a bit garbled 