Well it is completely understandable to imagine and dread that type of scenario, you're right, it does happen on occasion. what you have to remember is the more upset he sees you are, the further he will take it - this is all about him saving face and taking control. He's using the situation to feel big about himself at your expense...my guess is some deep anger issues against women in general, or similar.
Also, th fact he has reduced contact says a great deal about his actual motives. It makes literally zero sense - claiming obstruction and yet wanting less responsibility. Therefore all about saving face.
If you CAN, then try a different tack. What you need (and this worked with an ex of mine, very very well) is have a wall up inside your head. Feel strong - remember that no one in the world can force you to love, like or give house room to this man, ever again - there's no law to make you like him or even smile. All you have to do is comply with the law, and you and ds can do that atm so don't look too far ahead.
This man is a jerk. You have the wall up, right? So everything he does is going to be ineffectual. You tell yourself you don't give a shit, he is such a dickhead. He's also a bit thick from what you've said.
keep his motive in mind. pretend you don't really find him upsetting. Be the broken record - keep to your plan, 'please could you only contact me at such and such a time, on such and such a day, otherwise I will not answer' and just make it very clear where you are and what the rules are. If he breaks them it's his own problem.
You also need to be very clear on what you want. It sounds a bit confused - ie you want him to have 'good' contact with ds, but that's clearly, clearly impossible - he's actuing like a complete twunt to both of you, it's not even about ds to him. He migt believe it is but it ain't.
So drop the guilt - it's not your fault. I would go with the basic endeavour to make any contact he has as infrequent and brief as possible - don't tell him this of course! - and this will ensure ds sees you as the important person and ex will be less able to rock his little world, when he does turn up.
I would try and keep it out of court if poss. Be very polite, very pleasant but at the same time, cold.
You can laugh a bit at the ludicrous things he says but be genuinely unrockable..'really? I don't think that's such a great idea, do you?'! sort of thing. You are bigger than him inside.
Once he knows he doesn't frighten you, he will get bored. He's your classic bully who hasn't a clue what to do with his spoils. He doesn't really want them 
I maintained this attitude with ex. He started off being like 'I really want to get back with you, pleeeease trust me' ha ha (I was also 3m pg when dumped!) It was clear he didn't want another child/responsibility but was under pressure from his mum not to look like the bastard in all this.
i laughed off this bit anyway, and then he was cross because I named ds without asking his approval, and then he said he'd come and see ds at 2 weeks old. I agreed, keeping it very business like - no arguing, no aggro, just, yes, fine, see you then. Gave him the rope - he never showed up.
I didn't hear from him for 3 years - result 
Reverse psychology works well. If you are hard hearted yet polite they can generally find a way to blame you for the fact they can't turn up. As long as they can tell other people you were 'mean' to them or something, or stopped them coming (actually they are just afraid of your very calm anger)
they will often bugger off and you can then get on with your life.
Agree to most things, in theory, and i bet he won't bother. He's proved he doesn't actually want the access. Give him enough rope etc.