Don't do this to yourself, OP.
For me, the question wouldn't be 'Could I forgive him?' Because, in a way, I certainly could. I can understand and rationalise someone failing, falling short of expectations, doing a bad thing. And I could feel sorry for them, and lose my anger.
But that's not what it's about. For me, the question would be - Do I want to be with someone who could cheat on me, who WOULD cheat on me? And the answer to that is No.
No matter how hard the split and how much I had to give up, I know that I could never feel truly happy again with someone who had done this to me. Every special moment would be tainted. Every intimacy. To be thinking - to grow afraid of thinking, every time we had sex - of what he did. To fear that, if we married, the thing I'd be thinking of as we stood at the altar was THAT, and I wouldn't be able to look him in the eye. To fear that the moment our child was born, and we should be bonded, tearful, gazing at our baby and holding hands, that it would flick into my mind and I'd think for a moment 'You don't deserve this, and a part of me hates you.' I know I could possibly forgive someone I loved, but I'd never forget, and it would eat me up, and kill our relationship. I don't want the kind of life of buried pain which would be living with someone who had shown me how disloyal they could be.
Some people say they got past it. You're a couple of weeks in, and you think you can't. You're probably absolutely right.
This is without even discussing his attitude to it - which stinks. It goes without saying that anyone worth trying to get past this to be with would be crying, devastated, willing to do anything, feeling terrible, fully understanding why you'd speak to him like shit - simply grateful you'd be willing to speak to him at all. Your man? Fundamentally selfish, stupid, and entitled. He doesn't sound worth being with without this, to be honest.
And... there's one more thing, that no-one has picked up on. You say that you have been together five years, a good relationship, about to start IVF. Biiiig commitment. And yet - it's his flat and you'd have to leave? Now how does that work then? Are you honestly saying that you are about to undergo IVF with a partner you live with and yet your setup is such that you are simply effectively a guest in his home, it's all in his name, and despite the commitment you're about to make in TTC, you have no security at all? Well. Not good.
None of it sounds good.
You don't say your age. I assume the fertility problems are on both sides, but - you could undergo IVF as a single person, if it came to that. Leaving him and this situation does not necessarily mean never having children. I appreciate it's not simple at all, but it does not mean that.