My father had an affair when I was a young teenager, which almost caused them to split up. (Although I didn't find out that was the reason until my mum told me when was in the process of dying several years later. At the time my mum was effectively having an 'emotional affair' with what would later become her partner, I overheard a phonecall and reacted, and Pandora's box opened.)
I've forgiven both of them now. I think it was harder to forgive my father, because I was never able to have it out with him the way I was with my mum. And yet, perhaps that also made it easier, because he could never block a conversation the same way your father did.
I think what really shifted it for me was going through a period where I found myself, completely unexpectedly, drawn intensely to another man outside my marriage to my DH, who I love to pieces. Because of my family experiences I was absolutely terrified that I might cheat and really hard on myself, it was awful. Took some counselling to learn to forgive myself for having those feelings, and, though it might sound odd, it was in forgiving myself for them that I was able to realise that I could continue to not act on them, and step away completely, because that's what I actually wanted. (Ironically, the kind of men I'm drawn to are remarkably like my father - cold and ultimately unable to give me the kind of love which my DH does give me, but which I sometimes still struggle to receive because it feels so unfamiliar - and like it might be taken away.)
And I think in learning to have compassion for myself in this situation made it a lot easier to forgive him in the end. Because I realised that maybe, he just wasn't able to manage whatever turmoil was inside him at the time of his affair enough to step away. (Hurt people hurt people.)
That said, I think it's also true that sometimes we forgive for our sake rather than the other person's sake. We can choose to forgive because the anger that's still inside us ultimately hurts us the most (especially if there is no opening for direct repair with the person). I had to accept and forgive myself the pain that I still felt about my relationship with him in order to let go of that anger.
Sorry this turned into such a long ramble, hope at least some part of it may be helpful. And I hope you find what you need for yourself to let go of some of that pain.