When you say no and he asks why, say you feel this is the best way to go about things and the way you prefer it. After that, silence. Then it's up to him to either persist in asking (and you give him the same answer in a firm tone) or going on to something else and then returning to the topic. If you're in a public spot, he really can't raise his voice or get angry. This conversation could go on for hours, so if you think you're going round in circles or he's bothering you, look at your watch, warn him that you've got other things to attend to, say it's been an hour or whatever, and you're going. Then get your things together and leave.
The worst thing that could happen in this situation is ... ?
Try to shake off the idea that you control his reactions, but try to see also that he is restricted in his choice of responses by the environment where you are sitting, and also by the circumstances. He knows well that if he starts throwing his weight around, threatening, etc., he will possibly stand to lose contact with DS.
The idea that things will be done the way you prefer, or even that you have a preference that he has to take into account, is going to be hard for him to grasp. But he has no other choice. It is really hard to put your foot down, though, however gently, and no matter how much you are in the right.
Having a solicitor do your spadework on contact and visitation issues eliminates some of the hassle of dealing with the ex. They're very useful because they represent authority, and you do not. Trying to hammer out an agreement between yourselves will probably result in a lot of stress for you. Even if you succeed in getting a do-it-yourself agreement from him he will not respect what you have agreed, and will keep causing you headaches over any issue he can find, especially with visitation, because he enjoys this sort of treatment of you and it has been his habit up to now to treat you like this. And visitation is something that takes place on a regular basis, so there's always an opportunity for him to get at you unless everything is spelled out in legal terms, written down signed, and filed with the court.
Emotional abusers have to have someone to abuse. If it wasn't you it would be someone else, but for the moment it's you. Best to find a spokesperson and avoid presenting him with too many opportunities to do what he does best.