OP, ignore that first poster they have no idea what emotional abuse does to you. It's very well recognised that abusers deliberately goad their victims to lose it and shout back.
Remember, all you did was lose your rag and call him names under severe provocation after years of abuse. That is not illegal, the police and courts will be very familiar with your situation.
He, on the other hand, has made a death threat against you which you have a recording of and has physically assaulted you which you made a police report about (don't worry about the fact that you withdrew the complaint, it will still be on record and police DV teams are well used to this situation). Both those actions are most certainly illegal!
Like most abusers he is trying to scare you with empty threats. They are baseless. As you are the primary carer he has pretty much nil chance of getting sole custody. The most he could hope for is 50:50 and I bet he won't even bother with that.
The most important thing is to speak to the solicitor who handled your divorce ASAP. Ask about the possibility of getting an occupation order to get him out of the home. As there is a recorded history of DV against you it may be possible to get legal aid for costs. I am not legs qualified but your solicitor can confirm.
Also please speak to Women's Aid. They are used to situations like yours and can offer invaluable support.
He does NOT hold all the cards, please remember that. 