I don't want to get back with him, it is just access.
Yes, and that's exactly what they have a problem with, you allowing him access. This is really serious. If you go in there with the attitude that he's their dad, you need to allow him to see them, he wouldn't hurt the kids etc etc, you are at a very real risk of losing your children.
They see him as a risk to the children and if you allow him access, they will see you as a risk too. He needs to go through the courts for access and they may say he can have access in a contact centre. Or they may even say they think he's too much of a risk and can only send letters for now.
But if you go in to the meeting talking about his rights or how the kids need him or how you know he wouldn't hurt them any of that kind of stuff, you will be demonstrating that you are not able to protect your children from this violent man.
If you need to move, they may help you, and it's worth asking. My friend who was in this position found the system was very slow and she found a place to move to herself. I don't know if this was just her experience though.