I'm so sorry that you're going through this. Towards the start of my relationship I was in a similar position, with a woman my partner had previously shown interest in getting in touch saying she wanted to be with him. I believe - and so does he - that she only did this because she saw that we were happy and was jealous, so tried to come between us.
He did think about leaving me for her, because he'd liked her for a long time and our relationship was quite new in comparison, and while he was making his mind up I was so, so close to ending it for him - because I knew I'd be on edge about him leaving me, and so I was for a long time after.
Eventually he made his mind up to stay with me. I know I might sound weak for staying with him, but I honestly understood. This woman had showed up and played with his emotions - even though she had previously told him she didn't like him in that way at all - and I absolutely understood that after what he had been through with her, he had to be sure of his decision.
Three years down the line and he hasn't spoken to her and whenever we mention what happened (it seems to come up for one reason or another every now and then) he says he made the right choice and he thanks me for giving him the chance.
So my advice would be to let things calm down, and see where the land lies. I know my situation was very specific to me, as was how I handled it. I don't know all the details of what you're going through, but just consider the past between the two of them, how he feels, and what emotions it might have stirred up.
Of course it's absolutely up to if you want to leave. In many ways I believe that's what I would have told someone else in my position to do, and it's only with hindsight that I know I was right not to.