I hope you realise that a great deal of "the man he was" was not really anything great he was or did, it was him being there to allow you to project nice comforting things on. You've been in a long-standing habit of standing up for him, forgiving, glossing over, holding on to the positive (yes of course there were positives), fixing the love inside you at this object which was not necessarily worthy, just there. You fell into the trap that I think we all do, believing that the more you put in to a relationship the more you would get out, so you threw all of yourself into it and got back the reflection of love and companionship. You would have got as much from a teddy bear if it was your only company on a desert island. Only the teddy bear wouldn't have run off.
Basically, what I'm saying is, he seemed so lovely because he lived with you. You made him comfortable, you managed his moods, you encouraged his good behaviour, jollied him out of the bad. This is what any good adult-to-adult relationship should be like, only it should be like it on both sides, not one person all giving and the other all taking. It's quite appropriate that what you're clinging to now is a picture, because you've been clinging to an image for many years. The real person was mentally off somewhere else, planning his next shag, bullying colleagues, committing wardrobe crimes. Somehow you need to get yourself out of the habit of giving, giving, giving yourself into the black hole that forever absorbs but never gives back. Look in the mirror instead. That's where all the love comes from. And look at your lovely DCs. You gave them one-sided love for years because that's the nature of an adult-to-child relationship, but as they grow they are giving more back, bless them, returning love and support in their turn. That's what families should be like. Unfortunately one member of yours opted out of all that, because he is a selfish arse. Did I mention he doesn't deserve you?