It doesn't matter a damn whether his work is toxic or not, you have kids & a home that you need to continue earning wages to fund.
It's selfish and irresponsible to simply say 'Fuck it, I'm done!' and leave a job when you have a family who depends on your wages.
I'd actually be telling him that if he's not prepared to continue working until retirement age, you're getting a divorce so that the pension pot will be split accordingly, so that he can't deplete your share of it...
But if you think that's not what you'd want to do, then I'd be tempted to say to him 'You don't get to live off our joint savings/pension pot just because you don't like your bloody job! So either you get your old job back, get a new job pronto, otherwise I think we'll BOTH just jack in working until the retirement pot is empty, because I'll be damned if I'm going to carry on working while you drain our pension pot lazing about doing bugger all. We'll BOTH do bugger all until it's gone!'
Seriously, why the hell would he think it's ok for him to drain the pension pot, whilst benefiting from no working, but he thinks that you should keep working and not benefit from the pension pot at all??(because he'll have drained it by the time you're due to retire ffs!)
If it's good enough for him to do, it's good enough for you to do too.