He is always going to berate and have a go at you and blame you regardless of whether or not you show emotion, dont show emotion, let him get away with things, dont let him get away with things, reply to him, dont reply to him....its part of the overall bullying behaviour that tries to make sure he gets his own way.
So respond in the way that helps you most.
Remember that he is blaming you for things he's doing. This doesnt make it either your fault or true. He's responsible for what he says and does, not you.
We often fear that others will simply believe it when people attack us like this - that fear covers everyone from the neighbours to the judge! They don't and won't. People form their own views. That doesn't mean his family won't be on his side, they might be, but for others its different.
With the children it sounds as if you are struggling to keep the routines normal through this huge upheaval. On such upsetting days an evening together with you in a blanket fort with a big box of haribo or the equivalent and music or cartoons won't be the end of the world for them. Everyone needs treats, comfort and times without struggle, you and them. Its possible when hes not there. When you can, try and find these things
You are doing so well. It sounds as if its the truth that pulls him up a bit, for example when you said your biggest mistake was wanting him around. You've got this.