Ultimately it's not good that he's with you out of fear though....because it will only make him more resentful towards you.
He was having an affair after one year of marriage, which really begs the question did he want to be married at all.
I didn't realise you pushed him so hard on the second baby to the point of making his life miserable. Sometimes we are so focused on our personal feelings, but how would you feel if your son was in such a marriage in years to come?
You say he's a good father, so even if you split he would still be a good dad and you using your son as a pawn is ultimately harming your son, who could also become resentful of you in the end. Don't ruin their relationship, because he doesn't want to be with you.
Work towards amicable coparenting, which is in the best interests of your children.
Think back on your whole relationship and reflect on your actions. Who has been more enthusiastic about progressing the relationship...were your feelings for each other ever really balanced or was it more you into him.
What I see now, is him in fear and you in fear of him leaving, so you keep him under watch with no room to breath.
I remember a BH (betrayed husband) once said, he "wants a wife, not a prisoner"
He didn't want to have to police his wife's emails, he didn't want to ask for proof of where she was...it was tiresome, a job on top of his day job. He did for a few years after she cheated, then said he'd had enough of feeling like a Jailer. As it happens they are very happy and the trust returned.