Urgh - I don't really know what to suggest here as I have no experience of being in the incredibly difficult position you're in, flumpy.
It sounds as though mediation is not the way to go, but whether counselling would be any better or not? Hard to say... if you're both uncomfortable in those sorts of 'controlled' situations, I think the presence of someone else is likely to make things as awkward and difficult in feel as mediation has been.
If you're able to at least talk civilly to each other outside of that situation (where you are both safe, and able to leave if need be) then that might be the way to work towards a mutually agreeable resolution.
What your XP is saying now (regarding contact) may not turn out to be the reality - it could be that he'll decide that he wants to be a full-on parent once your DD is here. Or he may decide he wants no contact. Or he may stick to his few hours a fortnight. I would think that some contact (however small) would be better than none, but that is something you need to determine.
A friend of mine has custody of her 4 children, but when her XH has them (every other weekend) he spends most of the time out - and his new partner takes care of them (or at least as much care as she can, considering she has children from a previous marriage who live with them). The eldest of my friend's children is 17 and has said that she wants nothing more to do with her Dad, and doesn't want to see him. The next eldest has implied the same.
So, ultimately, the children are the ones who'll be calling the shots.
As someone who is inherently lazy, I honestly believe that there is no point making things harder in life than they have to be. Sometimes that means 'closing the book' on things and moving on. But rather move on and see what else comes your way than sitting still getting nowhere. Sometimes the secret to fulfilment is not to try, try, try until you get what you want (only to find out that it isn't at all what you thought it was), but to try, and if you fail, try something else.
Sorry, realise that's all rather rambling and philosophical. Please feel free to tell me to sod off! :)