One would presume, that when you got married, your sex life was pretty good. Or you probably wouldn't have got married.
Things have now changed (for you), because you have small children and you are knackered. This is understandable. However, his desire has not dwindled. He is now tied to someone, who makes it very clear that his advances (that were once met with warmth) are no longer wanted.
Neither of you is "in the wrong". It's understandable that you are tired. But it is also understandable that his desire for sex has not changed and he is upset at the continual rejection.
I would review the situation when your child is around 2 and/or sleeping thru, and you are not so tired. If you still don't desire him, maybe the relationship isn't going to last.
Like Marylynsbigsister, I have been on both sides. My first H sex drive did not change, but after children mine dipped. The longest we went without was 2 months. Eventually he sought sex elsewhere. In the end we split up.
I remarried and my sexdrive was, at this point, through the roof. The fact is, I found my current DH far more sexy that the first H. He lights a fire in me, that the first H just couldn't find. I still feel the same about DH, as I did when we met (many years ago). I fancy him loads. However, when he was bereaved, his drive dippped a bit and I felt rejected often. It is totally soul destroying.
Thankfully, this did not last long, and things are back to normal.
You need to find a compromise.
Often on MN, the man gets such hard judgment. That's not fair. He hasn't changed. You have. For an understandable reason, but I would caution against pushing him away so often that your little family explodes.
And in a passionate, loving relationship, 3 weeks IS a long time to not connect with your Partner. Maybe sometimes it can't be avoided. But surely it's not ideal.
I think in your shoes, I would set a day of the week for sex. Sounds non spontaneous, but needs must. Whatever your least tiring day is. For me that's a Weds, soin your shoes, I would say Weds night, half hour slot for regular sex.