DH has always been the one to initiate sex or intimacy and I have to admit that I would regularly reject him. He never hassled me and always respected my boundaries.
OP, how often was your husband initiating sex during the three years that you mention, and how often did you actually have sex? Because although you say that he didn't hassle you and respected your boundaries, at a certain point if someone is trying to initiate sex significantly more often than their partner wants it then it can become pestering.
I feel like half of the PPs have read a completely different description of your relationship to the one that's here. You describe a relatively short period (yes, in the grand scheme of a lifelong commitment like marriage three years is not that long) of less (not no) sex, during a phase of life where many couples have less sex (raising small children) while you were breastfeeding (known to commonly reduce libido) and parenting a child with SEN (so an extra pressure on top of everything else!).
What was your husband expecting during this highly pressurised phase of your marriage and parenting experience? Mine didn't really expect any sex at all for the first six months post-birth although in the event I jumped him as soon as I'd had the all clear the first time around and a couple of months on after the second child (both smooth pregnancies and easy deliveries). Our eldest didn't sleep for about the first four years though so after the initial enthusiasm once the chronic sleep deprivation had properly kicked in then yeah, we had a lot less sex than usual. Second time around my sex drive nose dived so we were having sex about once a month at my instigation (in opposite style to your husband, mine rarely initiated because he didn't want to pressure me at all).
Once my sex drive came back again my husband was thrilled! He'd been taking care of himself more often during our dry spell, although I didn't actually realise that until after the dry spell was over - I'd thought he was as knackered and unbothered about sex as I was! He's told me since that he'd have been up for sex much more frequently but recognised that I wasn't and was happy to wait until there were fewer demands on us from young children and I was better rested.
Our sex life now is very active and much wilder than before, the period where we had young kids seems to have been a hiatus in the general trend throughout our relationship of sex getting better with time. We talked about it recently and he pointed out that there have been other phases during our relationship with either very little sex or no sex at all (we've both experienced close family bereavements, we've been through job loss and some periods of depression, we actually didn't have sex before we got married at all which he says was helpful in terms of knowing how to do a relationship without sexual intimacy for a bit) and for him this was just another scenario that he rolled with in the expectation it would improve once other areas of life had gotten easier (which it did!).
So I guess I'm wondering what your husband thought the last three years "should" have been like, or what he thinks you "should" have done differently? Because what you describe sounds very normal to me, and in my experience sex ebbs and flows during a relationship for many reasons. However, it sounds like he is blames you for rejecting him "for no good reason" rather than understanding that your lowered libido was a normal, typical response to the situation you were in.
Apologies for the length of this, we've been talking about it recently and it seems like the bones of our situations are quite similar although the responses to them have been very different. But yes, in my experience it's totally possible (and normal, and expected as part of married life post small children or other relational challenges) to rebuild sexual intimacy.