I'm sorry, but if he's forcing your legs apart to do stuff to you, that's sexual assault and if you wake up in the night with him inside you, then that's rape. It's no less a crime than if you woke up to find a complete stranger had broken in and was raping you.
Because you don't have a high sex drive you're a) blaming yourself and b) thinking that this is what people with high sex drives need. Both of these are wrong (even if you told him you were never going to have sex again it does not give him 1% right to do any of this stuff).
My partner and I both have high sex drives and recently we couldn't have sex for 2 months because I was having various treatments/ops and my DP didn't so much as talk about sex. No gropings, no whining, no moaning, no pestering, nothing, and I know he was suffering. In fact, he wouldn't let me 'relieve' him as he thought it was unfair when I couldn't have any fun (the daft bugger!).
That is the behaviour of a good man, who respects and loves his partner and thinks about her happiness and needs as well as his own, if not above his own. If I don't ever feel like sex he doesn't even grumble about it. As everyone else says, your DH's behaviour has nothing to do with his sex drive and everything to do with abuse, power and control. Abusers have an uncanny knack for pinpointing their partner's achilles heel - in your case, having always felt bad for having a low sex drive - and slowly destroying them through that. Rape is about power and ego, whether it's a stranger leaping out at someone or a partner/spouse.
Do you have DC with this man? If things are this bad now, can you imagine what it'd be like to be forced into sex everyday whilst heavily pregnant, and raped whilst you are still healing from the birth? And what sort of influence would he be on the children? I'm sorry, but this man rings massive alarm bells on so many levels. Please, please, do something to help yourself get away from this situation.