I do think hes right that youre pursuing him unfairly. This wasnt grunt work, you had a family and made choices. You will never know whether your career would have been more successful if you didnt have them.
And tbh it does sound like he paid fairly, its just then he wasnt as well off as he is now. You cant say I deserved it then, his circs and yours were different.
But I agree, while I think hes told them the truth. I dont think he should have told them at all. I hope youre able to rebuild with them, but I dont think saying to them I feel "owed" for the "grunt work" will do you any favours.
If you do feel in hindsight you should have behaved differently over the maintainence and CB I think you want to base anything you say around that.