How terrible for you - this man sounds like a typical narcissist. It is going to take you time but you will manage to shake his abuse. Focus on yourself you need to block his unfounded worthless comments. Do some research on narc behaviour , it will start to become clear. It's shocking about your mom but sounds like he has done his sneaky narc work on her. You will get your self worth back but it will take time.
I experienced a narc and felt broken after - it took time for me to heal. I now know how they work , sneaky lying manipulation every one thinks they are wonderful until the shield drops.
You will need to build your self-esteem back , avoid all contact as much as possible. Narcissists only think of themselves- like to make everyone think they are perfect ie Dad , partner, employee - they absolutely cannot take any criticism at all......
At least he is out of your home , you have to work on the next phase - do not let him crush you anymore & do not react to any of his comments he is looking for a reaction - he will want to have the power over you.
As for your children he will probably slag you off to them ..... just ignore everything he says. A true narc is very typical and now you know what he is , do your research on how to deal with them.
I hope you have some friends who have seen through the rotten pig. You can and will survive this because you are stronger than a narc. They actually feel very inferior but cover it with their lies and over inflated ego......
Sending you positive thoughts for your recovery.