You need to get your 2yo used to him in the daytime and at weekends, and then she will be fine with him at night. (Yes, two out of three of mine have additional needs) sometimes you have to go through the hard times to make it better all round in the longer term. It might seem easier now for you to do everything for her, but for both your sakes, you need to get her used to him (and ultimately other people) Far healthier for both of you.
He sounds as though he has a nasty head of ostrich syndrome, and has buried his head well in the sand. I assume he has been getting worse since your 2 yo was born, and probably diagnosed, and you have sheltered him from the realities of parenting both multiple children and especially kids with additional needs.
Time for all of you to readjust. It's not uncommon (it's boringly ordinary) but you are letting him get away with it. How many of her appointments does he go to? How much of her care does he do? It's great to know that you can cope and manage everything, but, it's a really easy habit to get into, to do everything and then be exhausted. (And for fathers who are on the fringes, become more disenfranchised and check out completely, because they are not required as a parent) - you know, going to the gym and having a life and everything, while the martyrdom continues apace with mum)
Tim for him to step up, or you will all go the same way as the rest of the statistics, with relationships not surviving having a child with a disability.
Make him do it. I suspect by your insistence that you can't leave, that you have no real desire to. So make him step up. Or pick up the baby and disappear for the weekend and let him cope with the others. Preferably at least once a month, so he actually gets used to sole care. Then when the baby is old enough, leave him with the three of them.