Listen, this is the most important point, almost the only point you need to get:
You are not hurting him. It's not your fault. You're just stating your position and he has to deal with that. He's a grown man. You are being completely fair.
When you went into the relationship with him, you knew, as did he, that there were no guarantees. There never are. For whatever reason (you're not obliged to say), you don't want this. That's OK. You've tried it and don't want it. It's run its course for you. And remember, this isn't twenty years - this is just a handful of months. A few theatre nights. It really isn't as big a thing as he's making it.
Right, what you have to do is follow a simple formula and just cut it off: it's the only thing that will work. If someone isn't getting the message and is continuing to maul you emotionally, they really just need to be cast off. It's not your fault. He's the one who hasn't got perspective. And you don't owe him particular niceness just because he liked you (/was lucky enough to have your attention for a while, and kind of didn't make the grade - was to selfish/short-sighted and got in the way of your most important job, as a mother).
Send him one last text. Say something along the lines of:
Hi. I'm afraid I'm really not up for any relationship at the moment. I can't explain - it's just how I feel. I'm sorry and wish you all the best.
And then - this is the crucial part - don't text him any more
Don't respond to anything he says. Don't answer any calls. Completely remove that communication from your life. Because you've been fair, and he really needs to get the message.
I'm so sorry for you, as you've been through so much and had to be so resilient to get out of the big abusive marriage, and it's a bit like you flopped and this other guy was just in the right place at the right time (in his eyes). You flopped, gave in, relaxed, let someone in, being probably too exhausted emotionally to make much of a choice, and then found that what initially might have seemed like a harmless boost was actually becoming quite any annoyance and getting in the way of how you want to live. So you had to, yet again, try to get yourself out.
Do NOT feel sorry for either of them. I totally understand the Pathos problem - you end up feeling sorry for them because of their issues/failings/whatever, and don't want to be mean to them. This is in fact not going to get you anywhere - it will just drag it out, and spoil your life, and possibly also not be much cop for your DDs.
You're not being hard or unfair or mean - you're being clear and honest.
After the abusive relationship with your husband, you will have established a pattern of compliance which ultimately you developed to keep yourself and the children as safe as you could. Don't fight the ogre - pacify him and hopefully he'll soon just go to sleep and all will be OK. Well, this coping mechanism is pretty engrained, and you're still doing it, almost instinctually. The second guy sounds a bit weak, rather attached to you, and in some denial (not listening to you). He also wants something from you - and you don't want to do that, but you are automatically stressed by not complying. You feel it's because you don't want to hurt him, but let's follow that through ... you don't want to hurt him, because what does a hurt person do? They react emotionally, often irrationally and unpredictably, and you aren't quite sure what he might do. He certainly might come up to you in the village and ask you out (awkward, embarrassing) or worse - ask you why you don't want him (really very uncomfortable). And what do hurt people also do? They get angry, they go for the person who's hurt them. Somewhere, given what you've seen in the past, you're probably afraid. Scared of his attention. Wanting to get out of his headlights and let him go past, while you and your DDs stay safe and hidden.
You feel you've exposed you and them yet again to a person who might be angry with you. You don't want to hurt him because you don't want him to hurt you and your girls. This is too scary a situation, so you tell yourself you don't want to hurt him because you're a nice person and that would be unkind.
This gets you off the hook, and hides the fear you don't want to express, because it's too difficult, too painful, and also if others knew about it, that might make things more dangerous.
But this guy doesn't live with you. He doesn't have kids with you. He isn't anything like as deep in your life as your ex husband. He's a gnat. A midge. Brush him off. He is an annoyance. Put him in perspective. You wouldn't do this to another person. So have no truck with him. Don't engage emotionally. Just be polite, and shut the door.
Hey, you got out of the big awful thing. You did it. You have done amazingly and been very brave and strong. So it's no big deal, nothing, to write that text and forget about it.
And if he asks you out and you say you can't, then he sees you out - you know what? He'll get the message.
But just make it clear and then no asking out, no responding, no nothing. Free yourself from it. It's much easier than you think. Just do it.