Given everything you have said, I'm not sure trying to decide whether or not it was abuse is that useful.
The fact is that your memory of him is one of fear, not of love and security. My father was a truly shit father. Fucked off with another woman when I was a year old, saw us when it suited him, let me down more times than I care to count and was a general all round hopeless father. And in the 70s, no one batted an eyelid at this and it was considered completely normal.
But I was never scared of him, and my memories of him are almost invariably loving. In his unbelievably crap, damaged way, he made me feel loved.
Your father was clearly a very damaged individual. Whether deliberately or not, he made you feel fear, and fear does long term damage to the brain of children. Instead of feeling security, you had elevated long term elevated cortisol levels. That's the stuff that gave soldiers in WW1 shell shock. You had massive doses of it as a child. Of course it affected your self-confidence, as well as just about everything else in your life. Even if you learnt to overcome it and fight back, that damage did not go away, you just developed coping mechanisms.
It's good that he recognised what he had done to you towards the end, which is good but doesn't undo the damage done. The only thing that can undo that is massive amounts of compassion from you for the child you used to be, and the part inside you that is still that hurt child.