I’m a Christian and believe in forgiveness and the value of marriage if the other person is willing to fight and change themselves for the better.
I am a Christian too OP, and like you I believe in forgiveness. However forgiveness doesn't mean you allow yourself to be a doormat, it means letting go of the other persons throat to move on with your life. Forgiveness is about you, not the other person.
In this case your personal beliefs are your own, not your childrens. They have made their feelings very clear, and given he hasn't even attempted to try and repair bridges with them by going home when they come back, I feel this says all you need to know. I may be being unfair, and He may have worked on himself but one hot day doesn't make a summer.
He mentally abused your children, you put them first before, carry on with this. Your responsibility isn't towards fixing him or giving him another chance. Your responsibility is to the kids. Actions have consequences and in this case the consequences are he lost his wife and family. Tough. In my experience most abusers dont change their spots, or if they do it's down to years of therapy.
If you take him back it will cause irrevocable damage to your relationship with your older kids, and their relationship with their sister as they will take it, rightly, you are putting her above them and your need for a relationship with a man who hurt them, over them. They would be right.
Carry on as you are, arrange access for your DD so she still stills her father and make sure all kids are protected. There is nothing to stop him suddenly stopping taking his meds again and going back to square one when he realises he has what he wants.
As a christian myself I would never place a man above my child. He who hurts this little child hurts me, and all that...