Lack of self esteem has everything to with it. People with high self esteem say 'Sod that!' and walk away. That's what high self esteem looks like; walking towards things that are good for you, and walking away from things that aren't. It applies in all arenas of life, and for many, only manifests in one, so you may be super confident at work, and at relationships, but very unsure about parenting, for example.
It's not victim blaming to say that low self esteem is the issue, because it's not the victim's fault that they have low self esteem. It's generally that they haven't been taught to have high self esteem, from birth. The fault is with the abuser, no doubt at all. But the victim is responsible for their own happiness and fulfilment in life. It's not their responsibility to stop the abuser from abusing, but it is their responsibility, to themselves, to stay away from abuse.
We are all adults, and we are all responsible for our own actions. Abusers need to stop, of course, but the crucial fact is that generally, they don't. So, the victim is forced to play the only hand they have, and leave, for their own good.
Calling it victim blaming is like saying that when the house is on fire, the fire needs to just stop, and the resident is to blame for the fire if they don't walk away. There is a distinction between blame and responsibility that is often not made. The fire/abuse is external to the victim, not their fault, not their responsibility. Escaping safely is a valuable tool in their hands, and saying that it's victim blaming to advise them to use that tool minimises all the power they have.