I bet you left your ex with your child when you were living together, even if you just went for a shower or popped to the shop quickly.
Would you supervise your son at a nursey/childminder/playschool. No, they'd be telling you to relax and head off home and let them get on and do their job. Also who supervised you when you were a new mum with no clue how to be a parent? You have to relax.
A 15 minute 'chat' and coffee and then let him take him for a walk to the corner shop or park for another 30 minutes, but really does it have to be you there for that 'chat'?
Your ex should be seeing his child as frequently as possible in order to build up a strong foundation and good relationship. Certainly discuss if he has any concerns, and perhaps suggest your mother, his own mother, a sister or even a friend of yours are there in the background the first couple of times. This will be hard enough for him without you being there too. Don't allow the time with his child to be confused about the situation with you. SOmeone described it as a living berevment, that is exactly what it is like, and it can open up wounds and stor emotions and feelings up to see someone again, especially in their home setting where the loss of their family, family life and future together is even more poignant.
Think about what is best for your son. A quick transition into having his dad back in his life is probably going to be much easier for him than you. You said he's a happy child and his dad isn't actually a complete stranger now is he...
So after a couple of 'supervised' visits what are your thinking in terms of forward planning? And what is your ex hoping for? A gradual build up from an hour 3 times a week now, then to 2 hours at a time, then going up to 4 hours at the weekend, then 8 hours, then an overnight, then a weekend while continuing the mid week couple of hours...
What do you see the progress as being and how quickly?