A normal loving man would not make your anxiety levels rise just by coming home from work; indeed, they would drop.
But he's not a normal loving man, is he? He makes you scared. No partner/husband/boyfriend worth his salt makes you feel scared. Your anxiety is telling you something. It's shouting at you really loudly. Listen to it.
I bet that once you're away from him, it becomes so much less that you barely notice it.
You don't want your not-very-well baby growing up scared. Take her out of it. You will find the details work themselves out.
If you were to call the police and report the threat and intimidation of last night, then you will have proof of his abusive nature which you can then use to get a non-molestation order, get him out of the house.
He will have to pay child maintenance, so you'll have that coming in, maybe some benefits like housing benefit. It all adds up. Maybe finances won't be quite so good, but you and dd will be free to make happy lives for yourselves.
At the very least, call Woman's Aid today, keep calling them, and see what they say.