Also - what if someone says - I DONT THINK IT S A GOOD TIME
husband goes ahead
Wife feels too scared to speak up
What is that??
Assuming that this is all one event, where the wife isn't up for it and says so, the husband goes ahead, and the wife is too scared to speak up - well that's rape I'm afraid.
Even if you had given blanket consent to let him have sex whilst you sleep, you said it was not the right time on that occasion, therefore the blanket consent doesn't apply that day. I'm not sure legally how that would stand up, but no reasonable person would ignore what you said and continue.
Assuming you have not given blanket consent, then this is rape clear and simple. It doesn't matter that you were subsequently too scared to speak up. You did not consent in any way to this. Being too scared to speak/move away is the freeze response. When scared people fight / flee / freeze, and all are valid responses. Sometimes your brain decides that the best way to survive this is to freeze. And freezing absolutely does not imply consent.
This is why the threshold for consent should not be a coerced 'yes', a nagged 'yes', or a lack of response. The lack of response, lack of 'no' or fight can be because the person is too scared to do anything but stay still. The threshold for consent should be enthusiastic consent.