Have you had a open calm conversation with him about sex - both in general and as part of TTC?
I think there's a lot of pressure on both women and men to think that men ought to have a high sex drive. This can mean that with a man who doesn't can feel embarassed to talk about it and his partner can feel utterly rejected.
There is an age difference between me and my DH and we both have bouts of depression. He has a lower sex drive than me (though not non-existant, and he enjoyed it when it happened) and depression affects us differently (makes him less in the mood for sex, makes me crave physical closeness and reassurance). I used to get very upset and distressed when we didn't have sex. I hated initiating sex because the rejection felt horrible and it caused a few arguments.
We TTC for nearly 6 years. He genuinely wanted to start a family, I had no doubt about that, and he felt down about how difficult we were finding it. I have long irregular cycles which meant there were less chances, and meant that ovulation testing would have been prohibitively expensive!
About 18 months ago I decided to lose weight as it may have been contributing to my fertility problems, and even if it wasn't, I felt that fertility medical help would be more forthcoming if I was closer to the 'healthy' BMI range. I focused less on getting pregnant, and more on getting my body healthy for pregnancy. I lost 6 stone in about 9 months, which was about halfway to my target weight. I started feeling better in myself and began to accept what he had been telling me all along, that him not wanting as much sex as me wasn't down to a lack of affection or attraction to me, it was just his own mood and physical tiredness. So I got less 'needy' about sex as I didn't take rejection so personally. As I wanted to lose weight before we concieved, I was also less desperate to concieve 'immediately' more that it was a general goal we were working towards. Which probably took the pressure off.
Around this time, my DH went away to a conference for 5 days or so. We are rarely apart for this kind of time, and I think the combination of him being away from the stress of work, missing me and coming to some decision that he really wanted to concentrate on concieving (as I had done earlier in the year when I had decided to change my eating habits) meant that we had much more sex when he got home. We were more playful about it and tried a new position which is now our mutual favourite.
We got pregnant 2 months after his trip away. He reckons that we just weren't TTC often enough before. I think it's also to do with my body being more healthy (my cycles, whilst still long were shorter than they had been). Or we might just have got lucky - it'll be difficult to tell until we are TTC again.
I don't know exactly what I'm trying to say by sharing our experience, only that your DH may genuinely want to become a dad but just hasn't made the mental/emotional adjustment that it'll mean more sex, it must be difficult if its not something he is enthusiastic about normally.
I also don't think that the fact of his lower sex drive means that you are somehow incompatible. But it might be difficult if its something that you both find difficult to discuss.
I think you need to have a calm and open discussion with him both about starting a family and your sex life more generally. You need to express how important having a family is to you and find out if he genuinely feels the same. If he does, then does he know that you only have a small window each month to concieve (I agree with those who have said that we spend our pre-TTC days convinced that one slip will result in pregnancy, it can come as a shock how difficult it actually is!) and explain the biology. I'd also suggest that you explain that you can chart/take tests/etc to work out exactly when you are most likely to be successful.
Ask him whether he'd like you to inform him of these times so that you can target your TTC on the most likely times or whether that puts too much pressure on, and whether he'd just like to try and have more frequent sex and see if that works. I know a lot of people have suggested that its better to relax and not-chart but if he genuinely doesn't want to have more sex but does want to concieve, he might prefer to make sure that when you do its at the right time. I think its best to give him the choice and not make the decision for him, you need to work through this as a couple.
More generally about your mismatched sex drives. Again I think you really need to have an open discussion about how much you enjoy making love with him, that you'd love to have sex more often and that you feel rejected and hurt when he doesn't seem to feel the same. You want to find out if there is a problem, does he feel pain during sex? (no need for embarassment, its best to go see a doctor and get it sorted!) Are there positions/acts which he finds more enjoyable than others? Is it just that he is stressed out by work/family/TTC/etc? Does he find you attractive? Is he just not confident? Obviously, don't fire these questions at him, but just listen to how he feels about sex generally and your sexual relationship in particualar. Does he enjoy sex when you do have it? Is there anything he thinks he might enjoy that you could try?
Sex isn't everything in a relationship, but you should both feel satisfied by the physical side of your relationship as much as any other.
Good luck :)