Yes, get used to being alone - this guy will let you down, and big time.
If you honestly think that he will cancel having his child for the week because his new unpaid nanny (ie you) won't take on the brunt of the care, then that is one shit dad. One really, really shit dad.
So do an experiment. Say no to them coming over (ignore everyone on this thread comparing your DC to his, etc. etc. by the way - not relevant, you don't even live with this guy, even though you are having a baby together - that isn't the point of this thread). If he cancels on his child, then I honestly would dump him now. Really. Unless of course having a shit dad for your child is your dearest wish.
Sounds like this bloke is a classic cocklodger who realised he'd fallen on his feet so got you pregnant quite happily to - as he thinks - get his ticket to the table. So... he gets to go from doing the bare minimum for his child, to being able to host them overnight and have them for a week, like a proper dad - because muggins will actually do the parenting. From living hand to mouth to having a proper house all decorated and nice - because muggins will do the DIY, even at 36 weeks pregnant, and all Cocky McLodge has to do is sink onto the sofa with a contented sigh and survey the new paintwork... because he gives up after half an hour don'tcha know!
You have the sense to see it's a car crash now. You are clearly motivated, intelligent, and already have a child who you have presumably parented singly... why are you with him? Remember, having a man like this is never 'neutral'. He isn't just not contributing... he will be actively taking. Taking time and money and effort that could go to your children. Taking away your chance to find a functioning partner. Taking taking taking.
Have a good think about this, and start by telling him this will be an ideal opportunity to have his child alone for the week, in preparation for the huge increase in hands-on childcare he will be doing very, very soon. He might make your difficult decision for you sooner than you think ;)