When I first met my dh he had been divorced for three years and flew to London every Fridy to try to see his ds. More often than not he would return Saturday lunch to Scotland depressed because she wasn't in, ds had a cold, he was at a friends or dh was just a bastard and she wasn't allowing him to see him.
Dh hadn't ended the marriage, she did, she had an affair and moved with ds to London. I fell in love with dh and saw how this was tearing him apart, and that he was living in a bedsit whilst paying her £1200 per month. She was also claiming every benefit and dh had given her half the house equity, 50K. I told him he had to sort it out, get contact formalised, for his sake but more importantly for ds's sake,. He also had to sort out the money, or there was no way I was hanging around to bail him out/house him, in effect, I would be giving his ex my money. I was perfectly within my rights to say those were the conditions of our continuing relationship. And he did sort it, and it was the best thing he did, for everyone.
Your relationship started under different terms, I was nothing to do with their divorce, you were. So, a little bit of understanding must come from you. But, you are also entiteled to be able to plan for you and your child's life, and that includes contact.
Your dh must get himself organised, for all your sakes. Otherwise, you just may find yourself being another sad statistic.