Try and forget what's happened with contact in the past. Your son is having what seems like a good relationship with his Dad now. Don't be the bad guy by getting in the way of it, but do make the boundaries clear.
Don't engage in conversation in front of the children over this matter, and try not to loose your rag at your Ex. Deal with it as an adult should. I would write the Ex a brief and to the point letter (state facts and try and keep it unemotional). This should avoid conflict while discussing and to make your points clearly. Keep a copy of it in case you need it later so you can show you are being reasonable at court should it come to that.
If your Ex wants to vary the arrangements (have an additional overnight etc) make it clear to him that this has to be agreed at least 14 days in advance with yourself. Also be clear that it is confusing for all the children to attempt to vary the arrangements at the last moment and you don't want your son or daughter being brought into a discussion that should be between the two of you only.
Also state your point that you work all week and wish to have weekend time with your son.
Speak to your son and say you have discussed it with his Dad and that this is a matter for grownups to discuss and while he can talk to you about anything you and his dad must agree these things between you two. Also you will take his thoughts and feelings into account but any extra time with his Dad must be agreed at least two weeks in advance and you have explained all this to his Dad. Bite the bullet, show willing for your son, and agree to extra time one weekend if agreed 14 days in advance and see how it goes.
Does your ex have him overnight midweek? Could he pick him up from school one day if he doesn't already and have him for tea or even overnight? Suggest that as an alternative.
It's normal for a boy to want to spend more time with his dad as he gets older. However your Ex is getting all the fun time and none of the mundane school run, homework, uniform to sort out stuff by the sounds of it. Give him more of a taste of what being an equal parent is, or perhaps suggest every other weekend so you can both have quality weekend time together with your son.
Example Letter you could write:
Dear Ex
Further to our phone conversation [date/time] please be aware that I will not agree to any spur of the moment changes to arrangements with regard to the time [son] spends with you.
I do not believe it is fair to envolve [son] in any discussions we have over contact arrangements. It is upsetting and distressing for him. Please ensure all further discussions of this nature are not conducted in front of the children.
If you occassionally wish to have him for a longer time please could you discuss this with me in advance. As you are aware I work every week day and while I appreciate you wish to spend more time with [son] at the weekends it is also the only time I get the opportunity to spend quality time with him.
I would be willing to discuss the possibility of you collecting [son] from school during the week and having him until [time/overnight]. This would need to be on a specific night each week so that [son] is not confused by the arrangements and we can all work around it.
If this is not convienient perhaps you could have him for some additional days during the school holidays. Any arrangements should be agreed at least 3 months in advance of the holidays as I will need to make child care provision and book holiday time with work etc.
Please could you let me have your written response within 14 days. Otherwise I will assume you are happy for the arrangemenst to continue as they are in that you collect [son] at [day/time] and return him at [day/time] each week.
Yours sincerely
Follow this up with a further letter depending on his response... eg have not heard from you so assume you do not wish to vary arrangements and they will continue as [days/times] or if he discusses verbally Further to our discussion regarding the letter i sent you dated [date]... (then whatever you agree) or Further to your letter, I agree to you collecting [son] from school on [day] and returning him here by [time].... etc
Also any time he asks to have extra time as per the 14 day suggestion follow it up with a letter confirming you agree or stating why you do not agree on this occassion and suggesting an alternative.
If you need help with any follow up just ask
It's not easy being the reasonable parent, especially when he has mucked your son about in the past. But think about how it will look to your son if you act this way and don't always put blocks in the way of him seeing his Dad. Also think about how your actions would appear to a judge if this ever went to court. Your son clearly would like to see a bit more of his Dad, try and find a solution that works for you all.
Best of luck
Gilly