I felt like this with my ex husband - one of a few reasons we split, but not the main. I can't explain why, I have always (and still do) loved sex - I will say, affection toward him was never diminished though - I still loved hugging, hand holding, kissing etc...
I have a new partner of 3 years and I go through what we call my 'frigid' phases where we go for up to 8 weeks with no sex - I cannot explain it as I am so in love with him and very sexually attracted to him - I find the problem is gets a bit worse when I've realised it's been X amount of time since we last did it and it all becomes a bigger thing in my mind than it should...
Thankfully, my current relationship is a lot more healthy than my last and communication is free - rather than him continually trying it on, he just gives me a gentle verbal reminder (which is what I suggested, but normally I know how long it's been)..
I get frustrated with myself a lot too, because once I get going, I think 'Why have I withheld for so long, this is great!' - still doesn't stop me doing it in future, sadly.
There are lots of other things I do for him if I don't feel like actual sex - we really try to keep the intimacy going throughout a 'sexual drought' - so whilst he might go an extended period without me throwing him a bone, he knows I'm still interested in him in that way.
I think my ex-husband's a mistake was constantly going straight for the prize so to speak - nothing made me freeze up in utter dread quicker... If i even have him a hug or kids before going to sleep, he'd try it on...
Definitely have to discover what your issue is though - discussion and communication are paramount...