No you are not the bad guy - you've worked to maintain ds relationship with his dad. so on some level you trust ex with DS.
i get your concerns, I really do.
i also get not reporting it at the time - i didn't press charges against ex when i could have done as naively i thought he was getting help for his MH crisis which apparently prompted him to attack ds... not realizing it should have been left to cps to decide if he was "mad" or "bad" at the time...it would have made a big difference to what happened later with contact etc.and then a second crisis and him attacking dd.
but you have to be realistic as well
you could bring up the GP visits etc if you need to, if he takes you to court for child arrangements order - you will have ref number for your later report too.
you could speak to nursery safeguarding about your concerns. they can follow up with SS for you.
they can also look out for ds any signs of distress etc
but - all you have against this woman is some gossip and hearsay which may indicate she is a "bad" person.
you really don't know for sure.
you also have concerns that being in a relationship may trigger in your ex some violence, due to his past with you.
but that is all they are - concerns.
the reality is that until it [what ever it is going to be] actually happens there is little to go on.
you cant accuse someone of something they may do in the future...
if ex is happy to go along with your new arrangement then sure go ahead but your ds will be equally confused, possibly.
do talk to someone - your counsellor, the nursery safeguarding person etc. call SS and report a safeguarding concern if you wish but it might backfire... as nothing has actually happened [yet].
make a plan for what to do if something happens or DS seems distressed.
maybe talk it through with NSPCC helpline first, see what they advise.