If he's been like this before you were pregnant, and now he's like this when you're pregnant,with no real stress .....
I really really fucking dread to think what he'll be like when the baby arrives. Maybe you'll get a constantly sleeping, calm, chilled, undemanding, non crying baby you can take everywhere with no issues; but, tell you what, I didn't, nor did lots of people I know. It's been a marathon of sleep deprivation, interrupted sleep, colic, possetting (spewing up) crying, vaccinations, viruses, fever, teething (very hard going at times), tantrums (from long before two), interrupted meals, interrupted walks, stressful car journeys, tonnes of washing, crying, more crying (a girl I know said her son's cried so much, she could hear it like ringing in he'd ears even when they stopped), I could go on. It has tested myself and my partner to the limits of our endurance, patience, tolerance and calm over and over again, it has affected our relationship a lot, it had been harder than either of us imagined ....
I'll be brutally honest, I wouldnt fancy being you or your child in a house with this ban. You need to be either in the house on your own with as much help as you can get, or staying with relatives, with as much help as you can get.
He won't get custody due to having/making more money. Doesnt work like that. Whoever's the main carer (usually the mum) generally always gets it .. a d I'm presuming you'll be off on maternity and the main carer for your baby fir nine months or more?
Also anxiety etc are not reasons I could ever imagine social services or whoever removing a child from their main carer's custody. Lots and lots of people suffer from anxiety, depression etc.
Make sure you breastfeed (even if it's just combination feeding of breast feeding doesn't go well for you) .. in any case he doesn't have to know you're not fully breast feeding if you're not living together. You can use breast feeding as a means to stop him from getting overnight access/long stretches if he tries to go for it for as long as you do it.
What you also need to do is log his behaviour including the past stuff, so it's known/recorded that he's aggressive and violent. That's v important and will go against him in anything he tries to say about you/do to you.
The best way to do that