I did leave him a few years ago. Ended up living with all the children in the back room at my mothers for a year, because the council wouldn't rehouse us as we were "adequately housed" already! Eventually, I couldn't take it any more and had to go back. (Private lets are very few and far between here and the children need security, so it wouldn't really have been a good idea,even if we could have got one.) He was much better after we went back, because he never believed that I would leave.
TBH, I do think that he has AS, even my older boys have asked about that. My kids are his and he has another son from a previous, relationship in his youth, who also has AS. However, he is also compulsively lazy, not so much selfish, in a material way, more inconsiderate and kind of non-thinking.
The way I see it is that things are not so bad that I need to leave. I just do my thing and get on with it. I would not put the children through the huge upheaval involved in leaving unless things were diabolical, and they are not. There isn't even a bad atmosphere, in here, except if OH is feeling unusually paranoid, because I know what to expect, so I no longer react badly or expect ant different, really. So, there is not a compulsion to put my kids through an unmanageable level of change. It's not ideal, but it's not that bad, either. Re: the family thing, they come when he isn't here, sometimes, and we just don't tell him. Mostly, though, we go to them because my mother's health is not great and on top of that, they often look after my nieces and nephews while my siblings and their partners are at work.
Its not about trying to justify him or being on his side or anything else. It's about making the best of the situation. All my life, I have been in much worse positions than this, not of his making. An unhelpful husband is the least of my worries, to be honest.