If I’m responsible for myself then I’m responsible for my quite pathetic reaction to him and I’m ashamed of myself for that
This will be a huge part of your ongoing problems, and this is what I've been trying to get at. Yes, you are responsible, but you do not need to judge yourself as shameful, or your reaction to him as 'pathetic'. What can you do to help yourself feel that your responses to him were understandable? What will help you to validate your younger self? What if it had been your daughter in that same situation? Would you be telling her now, years after, that she should feel ashamed of herself for the way she responded to her abuser, or can you see how unfair, and frankly, downright mean, that would be? You are judging yourself negatively, harshly, and in a mean way. If you can find a way to see your actions in a more acceptable light, you will not have to carry such a weight, going forward.
It sounds like, when your mum used to have anger outbursts, you used to have to respond meekly, because she was your mum, so you couldn't just say 'Right, I've had enough, I'm leaving!', and that's what happens: we get conditioned to stay, and put aside our feelings of discomfort. That's what everybody does when they've been raised in a way where they had to put their own feelings to one side.
This is also what causes anxiety. Anxiety and a sense of needing to control everything comes from feeling like, if things go out of control, they will go beyond our ability to cope with them. The solution isn't to control things more, it's to learn to cope with things being out of control, better. Have you heard the allegorical story about the man walking on the broken glass road in bare feet? He tried for years to pick his way very slowly along the road, trying to get where he wanted to go, trying to stop himself getting hurt, trying to push the glass out of his way, trying to avoid it, and it was everywhere. Then he found himself some shoes. Voila. The problems didn't go away, but he changed his levels of protection, and everything was fine. He didn't need to control how much glass there was, or how it lay in his path.
Your anxiety (anybody's anxiety) is due to a lack of shoes :) Your world is no more or less out of control than anybody else's; but some people have stronger shoes than you, so they don't feel anxious, and you can get yourself some strong shoes too.