*I even think he agreed to the adoption so that my focus would be taken off birth son with younger son.
There I've said it. Am I mad?*
From what I've read about his behaviour towards your eldest son from when he was a very young child up til now ... No, you're not mad. It's possible.
Besides you're so used to questioning yourself and wondering if you're mad,big you're wrong - because that's what he's been telling you (and your sons) since you married him. He's a real nasty, head-f*cking piece of work. Abusive, gas-lighting etc. And he took advantage of you being vulnerable and battered (literally as well as emotionally) from your first marriage.
You keep saying you've made a mess etc. - you haven't. You went into a marriage in good faith and had a child (and adopted another) in good faith, thinking you'd found a good partner in a long term friend/acquaintance. And as an accountant he must've seemed stable, sensible etc.
You put up with his behaviour during the marriage because (it sounds like) you had little relationship experience, and what you did have was severely abusive. And also because you lacked self confidence, something your ex made sure to reinforce at every turn.
You haven't done anything wrong, you haven't made a mess of things. You sound like a good person.
You can recover now, work through it, and enjoy your life.
Out of interest, did you decide to adopt your second son because of fertility problems (one of them being your ex's lack of interest in ever having sex!) or altruism or ..?