At 9 months you should be OK as he can't really do any kind of mind gamey things with a baby anyway. So control will probably look like - him trying to gain access to your space - e.g. I didn't tell address to XP - met him in neutral location ie a park, when it became winter, he expressed "worry" that I was having to walk to the park with DS in the cold, so offered to pick up closer. I thought this reasonable so met him at the corner of the next street. He'd then watch where I walked from and every week he parked closer and closer until he saw the front door and then he just used to come straight there. That was unsettling. I never actually let him into my house, though, and I think that was symbolically important.
Other things he can do to "control" with a 9mo are that he'll agree to contact and then cancel or change arrangements at the last moment. Just ignore, don't ever schedule anything important that you can't cancel when he's supposed to have contact and don't let him know if it bothers you. He might push for more time quickly. If you don't feel it's fair then don't be hurried. He can build up from frequent shorter visits to longer, less frequent ones.
If you genuinely think he's a danger to the baby then get onto social services and ask for support such as supervised contact.
You could also insist he sees the baby at his parents' house if you think that would work and be sufficient.
XP used to send DS back stinking of smoke and chip fat and with bottles of strong ribena and fruit shoots. He was just one year old so couldn't tell me anything which had happened at Daddy's house and it used to make me anxious/annoyed but actually I wish I'd let all of that go. I used to get on at him about the drinks and it didn't change anything, it just made him annoyed. You have to act supremely unbothered about parenting issues and accept that it's their choice and only step in if something is actually bad enough to warrant stopping contact, which would be abuse or neglect. It's hard to give up that illusion of control, but ultimately when you're living with them although you might have more control over the DC's diet and exposure to things like smoking and adult video games or whatever, they are living with abuse. Even if it isn't directed at them they are experiencing it second hand. By leaving you're providing them with a sanctuary and safe space and an alternative experience of family life which is really beyond measure. Unfortunately, the reality is that a child with an abusive parent is always going to be affected by it (either by the abuse, or by the absence of a parent) but the absolute biggest and most important thing that you can do to mitigate that is to leave, and the earlier you do it, the more impact that will have to lessen the effect of the abusive parent.