I would leave him.
So much more that could be said, but it really does boil down to that in this instance.
Lying. Even if meeting as a friend, lying about contact with a significant ex. Deciding not to tell you, when as you say - he'd go mad if the situation were reversed. What does that tell you about this person's morality, loyalty? A lot. Just friends? Needed a shoulder while IVF was on? Doesn't matter one bit. Liar. Liar about the really important things. Don't let him tell you for one moment that this decision can be explained away. It can't.
Ok, so let's say they're not shagging and he wanted to stay friends but was 'afraid you'd go mad'. (that will very likely be one of the excuses). Well, does he think that's ok? Would it be ok if it were the other way around? (as you've said, no it bloody wouldn't be). Sooo - what that says is that he thinks that a good way to run a marriage is - if you want to do or be something that your spouse would find a. hurtful b. unfair c. disloyal - you don't discuss it and be honest with them, you lie to them. There's only one end point to that, and it's misery and divorce, or putting up and shutting up. Do you want to sign up for that? I wouldn't.
And the final potential excuse - he's found it soooooo hard dealing with infertility, your treatment, disappointment etc. the poor darling simply hasn't been able to handle it and needed somewhere private from you to vent/cry it out so that you wouldn't have to deal with his stress too. Fuck Right Off, baby! YOU are going through horrid treatment, the latter part of which he's PRESSURED you into having (fucking deal breaker alone - how absolutely shitty) he can't even be bothered to take the time off to support you (or maybe 'couldn't handle it', but that's not what he told you - ah, more lies) but its' fine and dandy to prioritise his feelings in dealing with it all. What an absolute selfish wanker of a man. More than that, who does he turn to - an ex. If he tries to say that one of the reasons was this, I would be out of the door before the sentence had finished... the thought of you going through something so personal, to be handled by you as a couple, but him turning to an ex partner to discuss it? Utter, utter dealbreaker. I know, if he says this, tell him you're glad to know that in the future, if you're upset about him not being able to get it up or that he wants a vasectomy or anything like that, that he'd be fine at you meeting up with an old boyfriend to discuss it - after all, another man would understand, yeah?
Anyway that's running ahead. Bottom line - you are married to a liar and a selfish, deceitful shit. I'm very sorry that your treatment has not worked but I would be looking at the fact that you don't currently have children to consider as one factor which makes getting away from this man easier. This is your warning - this is the husband he's choosing to be. Walk away.