I'm back now and have read all the posts. As usual, lots of sympathetic mumsnetters have chipped in. I hope you find you can stick with us, we are not here only for a crisis (but IMO we are rather good in them ).
I had a very manipulative and bullying ex who used my daughters as pawns to try and assert control over all of us. He often used to change arrangements on a whim. We never formalised the contact arrangements in court and I feel that was a mistake now. As it turned out I discovered after several years that he was exposing my dds to a porn business he had set up (I had no idea) and so my dh and I stopped any contact except in public with my dh to chaperone them. He then took us to court and we had to spend £10,000 defending our girls. We were successful and he now has no contact.
Obviously hindsight is a great thing and I now wish I had moved away because I don't think he would have cared enough to look for us. But of course these activities of his built up over several years without my knowledge and at the time I didn't think it fair to deprive the children of their father, whatever I thought of him. He never paid any child support, although when we first split he promised it. I wish now I had formalised the money side of things too because I never got a penny out of him. In the name of trying to avoid any unpleasantness, and because I was frightened of him, I hardly ever made a fuss about anything. When I look back now there would have been a great deal of money, and I particularly regret not claiming it in light of what he has cost us since.
With a formal court-endorsed agreement about when he is supposed to have contact, he will have to stick to the conditions or lose the right.
I would also make sure that he has to come to your house to collect your ds, that way you will not be making needless journeys and being stood up.
I have to say that your ex does not have a good track record. Two other children by two other women whom he no longer bothers to see? And he appears not to have learned anything from that. I think it a very serious business to deprive a parent of contact with a child but you might have grounds to consider this a fair course of action when you weight up the likelihood of your ex hurting and disappointing your son. It seems likely that he will build up a relationship then drift away as he has in the past, leaving your son hurt and disappointed.
All the same there is something to be said for allowing your son to know him so that your son will realise when he is older that you tried your best to give him the options. It would be hard for you both if he believed that you had kept a loving father away from him. It is so hard.
For you, as others have said, you need a break from your son, as we all do, especially with him having ADHD which makes it even more tiring for you. You need a support network. Could you start a babysitting circle with other neighbours who have small children? If you pay in tokens you get a night out for free. If you cannot leave your son in order to babysit, you can have someone else's child at your house for a few hours or overnight to redress the balance.
Where do you live? Once you post your whereabouts you are likely to find other mumsnetters posting that they are nearby.