Affair or not affair, depression or not depression - refusing to acknowledge you as a human being is abusive. and doing that in the intimate space of a couple relationship is going to seriously mess with your self-perception, your self-confidence and (yes, I know this sounds OTT) but your rationality generally.
I can almost imagine all the strange, contradictory thoughts that run around in your head. And they probably stay there because your intimate day to day reality - that you bounce your perceptions off to see what is "real" and what isn't - is getting seriously messed about with by your relationship with your husband.
An important thing - it isn't you that would be breaking up the family. He is behaving in a way that really has broken the rules of intimacy and trust that make a relationship a healthy and a good one. Regardless of whether he's having an affair or has depression. It's almost immaterial. The way he is behaving, treating you, and what he has turned your relationship into is just wrong. Bad and wrong.
I do think SGB had it earlier. I think he should try a separation, move out. It'll help you get your head together. the selling-point for him, I guess, is that it will help him think about what he wants - and he'll get a taster of what the "alternative to being married" is.
Frankly, I doubt counselling is going to work. He;s had plenty of time to take action about his behaviour - which kind of says he has taken action - and you are currently living with the result of what action he has chosen to take .... It's not nice, is it?
My other suspicion (and it's only a suspicion) is that the longer you give it, the worse it will get - by little, small steps. and the worse your mental health and self-esteem will get.
One life, my dear. You were not put on this earth to allow a mixed-up human being to practise their abusing skills on. On the contrary, I strongly suspect you are a loving person, and this is all going to stand in the way of you living your one life the way you are naturally inclined to do.