I'm going to chime in as I feel we're going to need the other side's perspective... I'm a male and in exactly his situation. We've been married three years, together for 5 and my wife isn't interested in sex at all either... she says she enjoys it when it happens and orgasms every time, but just has no desire for it.
It's tricky because from your perspective(s), you're not big fans of it, (asexual?) and when we ask it gets annoying, pushy, and as you've said, it becomes this chore with a negative feedback loop where being pushed makes you want it even less... which can then cause arguments.
From his/our perspectives, I understand why (for my wife it is hormones (thyroid), depression... just trying to cheer her up is a challenge, so I can understand why she isn't in the mood), and I'm sure he understands too. The problem is the little niggling thoughts that WILL creep in... sex and intimacy are usually linked with love... to not feel like she wants you that way will often make you feel worthless. The feeling of 'she orgasms, how can she not want it?'- making you feel like there's something else in the background.
There will be lots of people on here that will just jump on the "you're not compatible, this is a red flag, he clearly doesn't care for your needs, divorce him" bandwagon, as I've seen many times before... but I'm going to assume you love him, and like my relationship, don't want this to ruin things. I've told my wife many times before that I would rather not be having sex but be with her, than to have sex with anyone else and not be with her. I'm not going to lie though, it IS painful to feel rejected every day.
For me, I've just accepted that it probably won't happen and in time he might too. Is it sex you're not enthusiastic about? Can you placate with oral?... or just don't fancy any of it? Do you 'not help matters' by running around naked, sleep naked etc?- I know you should be allowed to, but that could be viewed as an unfair temptation. Talk to him and tell him directly the situation... I think you BOTH have to really think of some compromises if you want it to work. In truth, I can't help you... I'm living it from his side and it's not fun. It has led to my feelings of sex being associated with rejection. Then there is that pressure of being able to perform on the blue moon for fear that if you don't, she'll want it even less. The negative feedback loop of getting turned on and rejected results in occasional erectile issues because your body just gives up with a "what's the point" attitude.
This is not meant to make you feel pressured or guilty and I am really sorry if it has come out that way. I'm just trying to offer the other perspective. I'm actually really interested in hearing ideas/solutions too!..