@tiggytape & others have said what I think too - you would be 100% responsible for your potential child's safety 24/7 regardless of continuing a relationship with their father because of the established relationship he has with his mother and her partner and their minimisation of the abusers actions or potential actions.
I have substantial mental health issues due to family members being beholden to the person who abused me because he was so 'helpful' to them for other reasons.
He USED that feeling of gratefulness for admittedly good deeds so - were they good deeds or a way to leverage control? I think the latter. So I can and have forgiven my family but the damage is done.
I am left with a fractured, but better with therapy, relationship with my family AND (so far & I'm over 40) life long suicidal thoughts and destructive behaviours because that predator stalked my family and convinced them they needed him and he was a nice person.
He wasn't, he manipulated them as he manipulated me and he was a disgusting, filthy, abusive man acting like a good neighbour and 'friend' of the family.
You do not even have a child yet and so are in a position of power over whether you choose to stay in contact with this family and most importantly of all, whether you bring a child into that dangerous environment.
As an abused child and a single mum I have to tell you unequivocally that it's hard enough raising a child without inflicting on yourself the need to police a paedophile AND those that think he isn't a danger - it will destroy you even if you manage to do it nevermind the utter devastation on you and any child if your careful policing failed.
Don't stay with him OP, not unless he can see and understand and commit to full child protection.
I have forgiven my parents but I never feel 'safe' even now and it is criminal to inflict that in a child because it lasts (for me so far, even with LOADS of therapy) forever so you can choose something better for your future children by saying NO no contact, no minimisation, no setting a child up to think it's ok to excuse abuse.
I hate when people say 'crying while reading' it seems so me,me,me but fuck that, I am crying and it's because no one helped me, and you are walking open eyed into something that may kill your child. Because even nearly 40yrs on I still want to go to sleep and not wake up to remembering what was done to me and how that bastard manipulated my family to make me the problem.
I hope you (and your partner if he has he sought help to process) can have a good and happy life free from the black hole of abuse and all that surrounds it.