Oh OP. I'm so sorry to read this. :(
That isn't an affair, it's a relationship. It's not just sex for him, there's intimacy there too and for all you know, she knows chapter and verse about the infertility. He may well be suffering a blow to his male pride and thinking that if she got pregnant at least he's fertile.
I have a theory, from personal experience, that women sometimes know when things are not 'right' in their relationship and pregnancies don't happen. Whether that's sub-conscious or not, it was the saving of me. I had no difficulty getting pregnant when I married my now husband.
He's been with you for 7 years prior to your marriage and he made vows that he hasn't kept. There's no going back because, what could he possibly say to top that promise?
I would speak to friends on family in real life and confide what you've said here. Nobody who loves you will put pressure on you to put up with this shitty man. If you stay, you'll forever be looking over your shoulder and you shouldn't have to do that.
Your husband's cavalier 'disposal' of his affair partner that he maybe loves, is disgraceful. He's vile. You don't have children together and that may be a saving grace for you too.
I'm really very sorry. 