Have you seen a solicitor, even for a free initial session? Courts favour the status quo, and if you've been a SAHM and he's been a weekend dad, with all primary care delegated to you, it seems highly unlikely 50/50 will be ordered should the worst happen, and it reach court. I would imagine he's been told that, too. And mediation is by consent, by definition. If you truly don't think his suggestions are in your child's best interests, you don't have to agree. If he tries to strong arm you and you feel bullied, voice that. Say you feel bullied and he needs to back off. If the mediator is any good, they'll support that.
I think you need to be really honest about what your son's needs are, separating out your own, and your ex's, and then try to advocate for that. It could be that your child is picking up on your own anxiety about the new arrangements, or it could be genuine distress at being so unsettled, and I think you need to have a think about which you believe is most likely before deciding what to do next. It's certainly very suspicious that he wasn't able to put him to bed one night in the week before, yet now he wants fully shared care.
I'm sorry to raise this, but if your ex goes part time and has care for 50% of the week, then his child support obligations drop markedly, and so will any claim to continue residing in the family home until your son is 18. There are huge financial incentives, potentially, in this. That doesn't mean it is the motive - many men are devoted to their kids and can't bear to be separated from them - but can be a factor, too.
If the overnights have been extended to 3 and it is blatantly not working for your son, reduce them again. Nothing is set in stone in childhood and your ex may need to work up to that number more slowly. It should happen on your son's schedule and not the adults'.
I honestly do think you need some legal advice here. What your husband is asking and his approach doesn't really sound very child-centred.