My situation is slightly similar although in my case it was a second child we were ttc so perhaps a little different, but I will share anyway.
It took thirteen months to conceive my ds, and at the point we had tests the first time my h was told that there might be a problem with him. GP said that he would need to have another test in three months, but that month I found out I was pregnant so we didn't go back.
Fast forward two years and we decided to ttc for another baby. In truth we figured it might take a year to conceive again so for the first year I was fairly blasay. However when nothing happened after a year we once again went back to the gp to see about tests. I had day3 etc tests and these came back normal. However my h again had a SA and this came back abnormal. He was told that although he wasn't infertile he had limited fertility and would need to seek further assessment/help.
At this point he decided that he didn't want to seek any further help, and resigned himself to being infertile. We did however talk about the possibility of having IVF or similar to try to conceive a second child, and my view was that it was very much a rollercoaster which once entered would be very difficult to get away from. What, for instance, would I do if I had more embrio's and didn't conceive the first time? could I destroy further embrio's if I'd only committed to one cycle etc? On balance I decided that it just wasn't a treatment route I wanted to go down, and so we decided that we would be glad of the ds we already had rather than put everything on the line for another one.
However at this point I didn't feel ready to absolutely give up. I held on to the thought that as I had managed to fall pg naturally once this might happen again. In fact I even thought that as I'd given up it was bound to happen, after all that's what always happens, isn't it?
. It didn't happen, and after six years of not using any contraception etc I went away for a week, and suddenly realised that my life had moved on, and that the place for another child had passed. I came back and told my h that I was now at peace with no more babies, and that actually I didn't want to try any more, in fact I wanted to actively prevent being pregnant, and as such we went back to using contraception.
Tbh he was less on board with the idea than I was, but I figured that the age gap etc was now too big, and I also wanted to go back to work.
We are no longer together and I am in a new relationship. Had I been younger I might have liked to think of having a baby with dp, however I do think now that the time for babies has passed for me, and I don't even really have that urge to have a baby, given I would rather like to have some holidays and such now my ds is older.
If going through IVF and such isn't what you want then don't. There really is no right or wrong when it comes to these decisions, because what works for one doesn't for someone else and vice versa. And even if you're not totally overtaken with the idea of having a baby doesn't mean that not being able to have one doesn't hurt, but it may mean that you will find other things to fill that gap in your life.
And if things change in the next few years you can always change your mind and go down the assisted conception route if you want. But equally the time for babies may pass for you, as it did for me, and you may find yourself in a place in five, ten years time where the idea of having a baby is one you wouldn't want to contemplate...