your husband’s comments that he wishes everyone would go away & it could just be you & him don’t sound romantic...that sounds really unhealthy way of thinking personally.
On thing that jumped out at me is further back you said something to the effect of
“He’s too cowardly to tell his family the truth unlike how he is with me”
But he ISN’T telling you what’s wrong. He is still being a coward with you & not confronting or facing a supposed problem.
It sounds like he has thrown you under the bus when speaking to his brother that you were the problem & now is keeping up with this narrative to suit himself. That’s not love, or caring about someone reputation. He has slated you to his whole family who see super close & the “ride or die” “family first” insular enmeshed type. So he knows they will take his version as gospel without critically analysing his actual actions behind the words.
I also note you are the first person your husband has had a proper relationship with, which I feel is significant.
I personally believe your husband is not having an affair, but may be suffering from a case of “grass is greener” and dislikes himself, probably hates himself (as most toxic people do) and will do whatever it takes to soothe himself, including hurting the feelings of those around him who love him most.
I tend to think he does not love you deeply & needs a HUGE wake up call that his behaviour is unacceptable. He has labelled you a problem, with no apparant solution & refuses couples counselling - you must know that deep down he is refusing couples counselling because he does not want to share his real feelings & he would be pulled up sharply by a therapist for refusing to even tell his poor wife who has pandered to his problems for months what she has supposedly done wrong here.
So I think you have no real option here but to ask your husband to go back to his family. Let them deal with him 24/7 for now. Please stand up for yourself & show that this treatment of you is NOT ok
& even if he is depressed - well I have depression to (though bipolar type so different but feel can speak on depression issues here) - it does not sound like depression. He sounds highly manipulative, dishonest & a liar. Someone of weak character who is unable to look within himself to fix his problems & causing trouble within both his family & his marriage petulantly wanting to be the centre of everyone’s attention & concerns.
His family has obviously been hurting at being shut out. He must have known this would hurt them - but being cowardly won over
You feel hurt at having lovingly supported him through his self inflicted family dramas & now he has flipped the script & labelled you as the problem - but being cowardly to not actually discuss the problem - simply fed you excuses & lies.
He has done the same to you as his family.
THE PROBLEM IS HIMSELF
Please, consider a trial separation & take control of the reigns back - he is not the captain of your ship. I understand the concept of separation, even a trial one is scary & seems unfavourable - but right now you’re living in uncertainly, confusion, hurt and understandably resentment thrown in.
This isn’t how you treat people you love. If “just being yourself” isn’t enough & was such a problem for him that he caused a rift in his family & has said you contributed to his depression simply by way of your presence - Why is HE not wanting to be honest with you so you have an opportunity to fix this problem ? Why is he so happy now to stay when the problems remain? This means the relationship is still toxic & he should want to leave. But he doesn’t.
He quite frankly sounds incredibly toxic & this kind of chop & change / sneaky behaviour / stories that don’t add up & bold faced lies to your face are the hallmarks of someone with a personality disorder - not actual depression. Just know this. I think he has problems far beyond what you can help heal in him
He must be made to be out in the cold, face reality & only then will he have an opportunity to truly reflect on his behaviour - currently HE HAS RECEIVED NO CONSEQUENCES FOR HIS BEHAVIOUR. Every action has an action - you haven’t acted yet, you’re living in limbo, he needs to go.
HE should be the one demanding couples counselling too. It sounds to me as though he thoroughly (and quite selfishly) enjoys 100% of your attention all on him, without giving a jot about how his treatment of you has affected you.
Your husband doesn’t even care about your reputation within his own family....it’s so utterly self centred.
I hope you’re able to shake him up in some way so he can GROW UP!