I think not doing IVF is much much more likely to put strain on your marriage. You risk trying for another three years, still no conception, then you try IVF, it doesn't work, you are left wondering "Would IVF have worked if I had tried it a bit earlier at 37? Did my husband's reluctance ruin our chance?" Lots of potential for bad feeling and resentment there.
Or no IVF ever, no conception--and again, what-ifs, resentment, feeling that DH robbed you of a chance.
Try IVF --soon. Seriously.
It's not an easy road, but the vast majority of people do not have their marriages ruined by it.
If it works, you get a baby--and your chances are better now than if you wait another three years.
If it doesn't work, you will at least have closure on the issue. "We did what we could, but it was not to be. At least we know we tried, and there are no regrets or what-ifs. It's good that at least we have the issue out of the way. Now, let's just enjoy lots of sex without the bother of birth control and get on with planning that big exciting trip around South America" or whatever.