Oh, thank you, pickle, that is very reassuring. And not TMI - I can see how it would be that way.
phoenix - just to add to the throng of opinions, here's mine. Or rather, my anecdote. My lovely friend did IVF, but very sadly miscarried. She was in a relationship at the time, but a little like you - they didn't live together and he was emphatically not the 'father' in the picture with the IVF. She fell pregnant naturally with him a few months ago. It isn't the same as your situation as their relationship has evolved, and though they don't plan to move in together and it's not totally clear to either of them exactly how his role will play out, they are closer now and did at one point consider him moving in while the baby was small.
However, seeing their relationship does make me think how many relationships that aren't the classic man+woman+kids, actually do work fine. And you have negotiated complexities with your ex, too. So I am not sure I would say, don't do it. I would say, think about it hard and talk lots - because you can't talk too much, and if he can't cope with the talking, it's a warning sign. But it could be fine. Apparently (and I wouldn't know, as I know literally no gay male couples under the age of 60) gay men and lesbians do this all the time, and no one raises an eyebrow.