As another childhood sexual abuse survivor, I think anon has absolutely every right to deny this man any access at all. Anon, remember though that you are pg and feelings will understandably be running extremely high. I don't mean this isn't important, but pg hormones will make it seem terrifying and unmanageable but actually I think you will be able to do it. Keep talking to DH - it's excellent that he is divided and can see your side - that shows a lot of maturity on his part that he's not just out drinking or refusing to countenance the whole thing at this point. Keep telling him you don't think you can bear the thought of your child being near FIL ever and you need his help in coming up with some suggestions for how to handle the pair of them. Keep saying it - you haven't done anything wrong; FIL has; you have every right to feel this way; FIL has no right to see his grandchild if you don't want him to. I myself no longer speak to my dad and have made sure he doesn't even know that I have a DS. Do I feel bad about that? - no - he messed up and put himself in a position where he cannot be in our lives, his problem.
However. I do want to say that avoiding all men is deeply worrying to me. You are never going to know who all abusers are and they could be women - it's unlikely, but so what - if your aim is to avoid all abuse, then logically you're going to have to avoid all people.
Also, it will mean you end up with DCs who aren't confident on their own and don't give off confidence and will have picked up on your terrors, and that is dangerous, because one day they are going to be on their own, one day. Far, far better to know that danger is everywhere and tell your DCs (when you think they are ready) about the risk, about the sanctity and privacy of their own body and how they can tell you if anything uncomfortable ever happens - even if they are told it's a secret. That will give them power and even then if abuse does begin, it will be caught. Yes I know you can't bear to think of your own DC being abused but it can happen and you can't bury your head in the sand.
I was abused by my dad and another man. I don't trust my mum - but not because the abuse happened, but because she turned a blind eye at first and also (with the other man) let him into our home again after it happened. Your DC won't distrust and reject you if it happens and you react the right way and take action. Only if you don't. And many, many, many of us have survived it and are OK. I'm not saying sexual abuse is fine - it's evil and terrible - but as with terrorism, avoiding any risk at all costs is ruining your DCs' freedom for the sake of safety. We all need to be vigilant all the time and realise it can come from anywhere - including your own partner - so trying to make it 100% impossible is ridiculous.