Is he named as father on the birth certificate? If not, he has no legal rights. You have a responsibility to keep him informed of his child's progress, but he has no automatic right to see her.
He can, however, apply to the court for parental responsibility and access. They will decide if it's in the child's best interest to grant it, and in most cases they do.
If he is determined to play a part in her life, there isn't a lot you can do to stop it. But there's all sorts you can do to manage it. He's a long, long way away from having your daughter on his own, let alone overnight. Years. Supervised contact would be the way it would start, and if you can't bear to be part of that supervision, then it would be via a contact centre.
It would be better if you took control of the situation - better for your frame of mind, better in the eyes of the courts should it get that far. I'd write to his solicitor and suggest mediation sessions. And I wouldn't be surprised if it all went away at that point. My sister's ex backed off at mediation. My ex husband wasn't keen when he realised they weren't held at a time to suit him. I don't say that you should suggest mediation to be awkward - I genuinely believe that, after 4 years and the way he treated you, that you will need third party help to arrive at a solution that's right for all concerned. The fact that 90% of men can't seem to be arsed is just a bonus.
I would feel EXACTLY as you do. I would be outraged on behalf of your new partner too, who truly is Daddy. If your ex isn't on the birth certificate there's nothing stopping him applying for parental responsibility right now, with a view to adoption in the future when the situation with your ex has died down. Try not to worry.
But do seek legal advice. Take control of the situation. Let your ex see that if you are doing this, it will be PROPERLY, with your daughter's best interests at heart, and in a way that allows him to demonstrate that he is in it for the next 20 years, not as a way of scratching a curiosity itch. Maybe send some photos of your daughter and a little bit about her with the solicitor's letter, in case that sorts it out.