I was born "into" a mother and baby unit. So were all three of my sisters. My mother also has bipolar, and when she got to 38 weeks in each pregnancy, she'd start to become erratic. She wouldn't notice it herself, so it was judged safest for her to be placed there.
We visited with my dad, either once or a few times a day, and it helped my mum a lot. She got a lot of time to bond and look after her little one, whilst she was being monitored. She wasn't "locked up", because she could walk around with baby in the pram, and we got to go out for longer periods before release.
For her last baby, she refused to go into a mother and baby unit. She discharged herself in the middle of the night, the day before she was due to go in. Social services were heavily involved after that, and did check-ups well into school. And of course, she ended up in the mother and baby unit anyway.
Please think about the best plan for you. If you are worried about family time, ask the psychiatrist to arrange times that your DH and older child can be with you. They'll help you feel comfortable and happy, if you ask.