How would your XP react if you told him that you have more than enough reason to suspend contact for a while, and if he doesn't dramatically change to follow explicit rules or good parenting guidelines that the contact will be permanently stopped.
Ask him what would HE consider a refusal to stop smoking when a child is highly prone to wheeziness/croup to be other than harmful/neglect/abuse?
What would HE think of a person seemingly perfectly happy to sit and watch a child struggle for breath, but still smoke in their presence and NOT call for help from anyone?
As I understand it, some BPD sufferers can live a life of drama, and if forced to accept that they will lose it ALL unless they get off their arses and COPE that sometimes they just can have a fair stab at sorting themselves out. I'm not saying cure, but if the drama feed is removed, they sometimes can just get on with stuff without creating the incidents.
the other thing to say is that clearly overnight contact can't be repeated, so the holiday plan is totally out of the question.
Remind him that the day that he proves that he CAN cope, that you WILL allow him all the appropriate access possible, but it's UP TO HIM to make sure that HE ups his game.
Where his mother is concerned, she is AS responsible for her behaviour. If SHE can't 'get' the seriousness of the situation, and HE can't make her, regardless of ANY improvement HE makes, your DC will still not be freely accessible to situations and people that are harmful.
Lay down the law, set the boundaries and state that you will not be returning to mediation, as there is no need, you are not interested in his words, only his ACTIONS will do. He just needs to step up, and if he needs help from you/GP/anyone to know that it's OK to ask for it.
Reassure him that you DO want contact, but that HIS choice to behave in this reckless and unthinking manner is harming your DC and you can't allow that, any more than he knowingly would.
Don't let him lean on any diagnosis here, he has choices, he can ask for help.
Ultimately this man is NOT currently a healthy person to be around, and if not the father of the DC, you'd not be trying to maintain contact. The fact that they are related by blood means NOTHING, the unhealthy situation is there.
If a person, either stranger or relative, harms/hurts/neglects a child, that child needs protecting/shielding.