But he doesn't need a Saturday job?
Presuming that he's actually done the maths, he clearly thinks you can afford both for him to go and do his activities, and for him to be paying £4 p/h for someone to be looking after your son while he does so. I'm going to presume that he knows the finances of your household, and that this is true, because that's a whole other ballgame.
Therefore he can go and do his hobbies, but you can reduce your hours so that you are looking after him, rather than the childminder. Which is what you said you wanted.
I get the feeling you object to this without really knowing why, though. Do you just not want your son in any type of paid childcare?
If you want more time to indulge in your hobbies, book a morning with the childminder on one of your days, too. Your son will be happy and well looked after, and get to socialise with other children, and you can enjoy yourself and then pick him up and spend the rest of the day with him.
But like I said, from your comments and your argument with DH, it's really hard to see where you are coming from.