i don't know what to say, except to share my experience (ie, to talk about myself, as usual). my current counsellor is pretty good, but she said to me that i seemed to have 'got over' having a poor relationship with my mum. it isn't true. you never get over your mum not loving you, no matter what age you are. and the same for dads.
so there are you, making a party and wanting it to be nice, trying to include everyone, and he doesn't love you and your beautiful, precious children enough to reply to his invitation. that's how it feels to you, i'm guessing.
he should reply. he might not have the same social expectations as you - having been invited he might think all he has to do is turn up.
oh, and when my daughter had a baby, i was in a state of terror throughout. before, terror that she wouldn't get chance to try for a baby, wouldn't be able to conceive, would split up with her husband, would die in a car accident or of the chest and kidney complaints she couldn't shake off; during, that the dreadful sickness would harm her or the baby, or because she can't swallow properly that she'd be sick in the night and choke to death, or that she would lose the baby, or that she would die in childbirth. terror. i couldn't make her safe. in the end, she survived, but it was a close thing - if the ambulance hadn't moved so fast, or if there hadn't been a full team already assembled at the hospital, she would have died. i'm not saying he feels that but he might be really uncomfortable with you being pregnant.
your children love you. if you can love them and show them how things should be done, you will break the cycle and no-one else will have to feel as you did when you posted. good luck.