I remember reading about a study somewhere that categorised people into high or low sex drives and looked for common motivators & patterns within those groups.
Those categorised as 'low' sex drive needed to be in a good mood for sex, not stressed, tired, hungry, or otherwise uncomfortable. Sex for them was an activity that required a certain amount of energy.
On the other hand, for those categorised as 'high' sex drive, sex was a stimulant. It woke them up when they were tired, relieved them of stress, lessened pain and discomfort, and generally perked them up however they were feeling.
For someone who's in the second category, like your husband, it's impossible to imagine not wanting it due to tiredness or stress. It just doesn't work that way for him, the desire is always there and it hurts him to hear that yours isn't, and you don't feel the same way. He's not being selfish. He's simply hurting and feels that your feelings aren't the same as his.
It's like if he told you he wasn't in the mood to cuddle, or wasn't in the mood to say 'I love you' after you said it to him. You'd feel hurt and rejected.
I don't think there is any way he can really just 'not feel disappointed'. He's just got to deal with it, and you've just got to try not to feel guilty about it
The bottom line is, he can't help the way he feels about the rejection any more than you can help whether you're in the mood for it in the first place.