Nasty situation, and I can see very easily why you'd be thinking about walking away.
The question is - do you actually want to be with this man anymore? Is there enough left for you to willing to work for it or have you just had enough? If there's no respect or affection left for him and you don't ever see yourself being able to get past his behaviour these last few years then maybe it is time to go.
Because practically speaking (and only practically!) you and the DC would better off alone. You're already doing all the work and paying all the bills - not supporting him would leave you better off, not worse. He is, in very real terms, a dead weight and a stress causing dead weight, at that.
In terms of your relationship with him, it is not surprising that you aren't 'affectionate'. Why should you be? He's giving nothng at all to either you or his family. When was the last time he showed you affection or understanding - not sex, that's utterly beside the point here - but support, emotional warmth, commitment to your family. Care about your day/week/month. When was the last time he asked you how you were feeling, made you a cup of tea, let you have a lie in?
He's set himself into a pattern of behaviour that's not compatible with your life and that, at least, needs to change.
As for the DC... if you're worrying about the impact leaving may have on them, well, I know a relationship with their father is important, but the only one they have now is very negative and the home environment must be dismal. It cannot be worse for them to only see him sometimes (or even never) than for them to see him only ever completely uninterested in them to the point of being asleep through their birthday?!
I think you'd be completely justified in walking away. And I would, unless your husband starts making some serious changes in his attitude.
If you do decide to stay, set a time limit for how long you're willing to give it. Then write a list of all the things you feel need to change, so that you're clear in your own head. You don't necessarily need to show him the list (although do, if he won't sit down and talk) but you do need to say to him 'I'm thinking of leaving but can we talk about it first.' If he won't even discuss things, you have your answer, because things can't continue as they are.
I'd suggest that, at the least, you need him to do the following:
Stop keeping his current hours. He fits into the family schedule, not the other way round.
Start contributing to the house. He should be doing the bulk of the housework, because he's doing nothing else.
Start contributing to his children. If not in terms of childcare (which imo, should be his responsibility at least at weekends) then at a minimum in terms of attention. Sleeping through important events is a no-no.
See his GP, if you think he's genuinely depressed and not just lazy. If he cites depression for why he can't do anything else, then this really is a make-or-break. Say you'd be willing to work with him to get treatment/get better etc, but don't let this become a 'get out of jail free' card for why he's not pulling his weight. You have yourself and, more importantly, your DC to think of.
Start actively looking for work, and take the first suitable job that's offered, unless he's willing to become a full-time SAHD.
Hope that helps - apologies for length!