Hi OP, just sat here open mouthed at the similarities of our stories, from shouldering the financial burden, to having to be the responsible adult in keeping a stable job, deteriorating mental health, to being punished for being a working mum and losing custody of a child 
I'm so, so sorry you are going through this
Everyday you just get on with your life with this little niggle that you can ignore some days but others eats away at you.. the niggle that you're being taken advantage of by someone who should be helping you support the home and family you both agreed on!
All I can say, is trust your instincts sweetie. You're in a great place if you're starting to realise the position you're in, you can start protecting yourself (seeing a solicitor, getting paperwork together, cementing your position as main carer- document everything- being a SAHP also means making doctors appointments, planning holidays, birthday parties, teachers meetings, involving children in extra curricular activities- youve done this before if you've been through court, now YOU know what cafcass puts in reports.. does he do these things??)
Maybe, see what happens if you stop paying the IVA, his reaction will help you decide if your instincts are correct. If it's the kick up the bum he needs to contribute, brilliant- you have something to work on for your little family... but, if he's still not bothered then you'll know what you need to do.
I wish I had appreciated and respected my self worth years sooner, it's a hard slog building my esteem and confidence back up!
You can't living the same day over and over then expect a different outcome.