I'm only 29 and have 2 years of TTC under my belt following an ectopic pegnancy resulting in fallopian tube removal. Lots of people have said how young I am, how much time I have, not to stress, to just relax, not to worry about getting tested yet etc. Fortunately I didn't listen to them, as having been referred for IVF I've just found out that as well as a missing tube, and a potentially dodgy remaining tube, I have AMH levels of 3.7 indicating poor ovarian reserve. That level is in the mid range of what they would expect for a 40 year old. I am fit, slim, healthy and active, with no health problems, outside of fertility issues. I need to get cracking, I don't have time to waste and there is a strong likelihood IVF won't be successful. That's the reality. It's not a silver bullet when all the other options are exhausted.
If you're worried about the effects of TTC or not, IVF or not, on your mental health or your marriage, think about getting counselling specifically with someone who is used to counselling couples going through fertility struggles. Hopefully they can help you think through some of your feelings and help you and your DH communicate about what is a priority.
If you decide to stop trying to conceive for a significant period of time, and there's nothing wrong with making that decision, only do it if you are prepared for that to mean that you may not have your own biological children.
I really hope you can find a way to come to a decision with your partner that you both feel comfortable with now and in the future.