If your father worked, then presumably the time he could spend with his children was limited.
If instead of spending that limited time outside of work with his wife & children, he spent his time with the OW then of course you have the right to be pissed off.
Depending on your age, he potentially missed out on story time with you, helping with homework, making you dinner, playing games with you, parenting essentially.
He also made life for your mum much harder. Leaving the bulk of the parenting to her, doing less than his fair share around the home, making her feel jealous, suspicious, unappreciated, neglected, unhappy. All of that would have also affected you, either consciously or subconsciously.
You are absolutely right to feel betrayed. In short, he prioritised his dick over you and your siblings.
He caused feelings of anger, betrayal & hurt in your family.
The fact that he thinks you weren't affected implies he's a narcissist as well.
Take some time out to process this & ignore his ridiculous statements about you not being affected, that's just wishful thinking on his part.