I totally agree with madamezthe problem comes when there is a mismatch in libido. The sex advice columnist Dan Savage probably reiterates this once a monthif you don't want sex and your partner does, you have to expect they'll go elsewhere.
My exH had no sex drive at all. For a few years, he made me feel unattractive and miserable, telling me I had something wrong with me and I was a 'nympho'. I wrongly assumed that because he was a man he had to have SOME interest in sex. I started pressuring him for it almost daily, even when I wasn't in the mood, knowing 99/100 times he would say no, all because I didn't want to miss that one time he might lower himself to sleeping with me. At one point he suggested I just sleep with someone else. I was completely shocked, and he took it back and said he didn't want me to do that, but it was too late.
Eventually I had an affair with the first man who made me feel attractive. ExH and I went to Relate and he finally admitted that he'd pushed me away. We went to sex therapist. Nothing really changed. I wanted children. He said he did, but you don't get babies by having sex once every two months, not when you've got PCOS. He also slept with a prostitute so I would 'see how it felt to be cheated on' and so he could 'try things' he said he couldn't try with me. My only response was 'did you like it? Did you learn anything?' His final offer was that I could have sex outside of the marriage, but only with a male prostitute. This was because he felt it was the only way he could be sure of never coming face to face with someone I'd slept with. I declined the offer.
Now because of the lack of sex, and because of the neglect in other ways, we are finished. I have a DP (due to be DH2 next year), and we aren't exactly breaking the furniture every night, but I'm happy. I don't have to initiate all the time, and I can say no if I really don't feel like it, without worrying that I've passed up my quarterly opportunity.
So yes, your partner doesn't have the right to demand sex if you don't want it, but at the same time, you don't have the right to demand that your partner is celibate if they don't want to be.
tbotb, you shouldn't feel ashamed for having those thoughts! If you're getting no intimacy of any kind in your marriage, that's what happens.