It's not the right of a mother to bestow fatherhood on another man in order to make a new family you like better, even if it's for the best of motives.
I wish I could wipe my stbxh right out of DS's life, he's been a terribly damaging influence and DS deserves so much better. But, I wouldn't be able to sever the actual biological & legal link he has, just because I wanted to.
There is a huge difference between what feels right from your own perspective, and what is the law.
By the way, it's not easy at all to withhold access if your child's father takes it to court. And no, a very limited amount of 'emotional abuse' to you after the birth wouldn't be grounds to deny your DC a relationship with their father. I say limited, as you say he hasn't been around hardly at all since the birth of the child, so presumably not really much opportunity - not meaning to minimise if that's not what happened.
Sadly, even if there's been a huge amount of abuse to the mother of the children, over years, which has been documented and evidenced, it's still not necessarily a cut and dried case. Contact could be through a third party doing hand overs or a contact centre etc.
Courts separate out the relationship between father and his children from the adults relationship difficulties, unless it's clearly & directly one parent using it to abuse and it's damaging to the children.
In my opinion the bar for this is set too high (and something I'll have to battle at some point) but at the moment, it's the reality.
Parental alienation is becoming something the courts are aware of, so I'd also be careful to make sure any future actions on your part can't be seen in that light.