Really depends.
DH has given blanket permission (and repeats this blanket permission periodically), but what usually happens is he's waking up by the time it gets beyond cuddling, kissing, stroking. If he was drunk or sleeping and unresponsive to the initial stages, then I would stop there (despite permission).
If there is no blanket permission in place, then it's absolutely assault.
If there's blanket permission in place but you've fallen out, then I'd say assault too because I would assume the consent is conditional - that if it's something the other person probably wouldn't consent to at that time if they were awake, then it's out of bounds. So that would include going through a bad patch/a fight/other person being particularly tired, or ill.
Personally, even with blanket consent, I wouldn't be willing to do this with someone who wasn't responding at all, but that may be different for someone else, and requires detailed negotiation.
I, OTOH, have not given blanket permission to DH for the other way round, and that's fine.