Bloody hell. What a mess!
First off, you have my sympathy as my DH had the same problem, and everyone thought it was me! Used to boil my blood. I came to understand that it was both of us, even though physically it was him.
Anyway, my DH and I managed to keep it all private, (nobody told anyone, as its a personal issue, and also i didnt want any labels, as things change) and underwent three years of scans and trials, until we found out where the major problem lay.
My DH had less than 5 % viable sperm, and those that were alive, swam in circles, if they moved at all, and then promptly died. 95% of them weren't viable, meaning they were badly formed, short lived and couldn't swim.
We were refused IVF and ICSI and IUI as the sperm weren't viable. We were offered it with donated sperm.
But then an amazing thing happened. We conceived!
For two years prior to that, both my DH and i ate organic food rich in zinc and minerals, took solgar suppliments, and were very fit and healthy. I took up yoga, he avoided tight underwear, and saunas ( the usual) and mountain hiked every weekend.
I went for regular scans to see when I was ovulating, and one afternoon my egg was seen leaving the ovary, so i rushed home, called dh and we had a cup of coffee together, and did the deed. I did all the usual elevations afterwards etc.... (romantic!)
Hey presto, we conceived! My egg was leaving my ovary, and met his 5% sperm in the Fallopian tube.
It only takes one superm, so don't write it all off yet.
My feeling is that you aren't infertile as a couple. But that you might want to consider changing your lifestyle, to optimise your chances.
Or you could get rid of your DH and his blabbing, blaming family!!
In seriousness, I think you need to have a chat with your DH, and tell him in no uncertain terms that you feel hurt and betrayed with his false revelations. Fertility issues can have a devastating effect on a couple unless they are honest with each other, and the blame game isn't played.
Also I think you need to have this discussion mediated by a counsellor, as my feeling is that your DH is being a sap and will minimise your feelings about this. He's already playing the poor me card with his parents, and the blame game.
He might even be receiving advice from his family to get shot of you, as "you" are the problem......
Best of luck to you both. And don't give up hope!