He said he wouldn't act like daddy while someone else is with his missis. I wonder if he suspects anything, or is he just being his jealous self.
It's probably natural to dream about what might have been if you had chosen someone else long ago. I sometimes wonder what would have happened if I had carried on going out with a boy I used to know when I was very young. Or about other 'chances' I had. But if they turned up on my doorstep tommorow, I would not want to be with them, as apart from anything else I need to sort my head out first! And make sure my dcs are ok.
I think you should treat whether or not to be with this old friend as a completely seperate matter. If you leave to be with old friend, your HP will have massive sticks to beat you with after you have split.
You need to be on your own first. Not go straight to another man.
You need to heal from this abusive relationship.
You need to be independent again. Before you get together with another man. And then you need to be sure your home is in your name, not his. If you end up having a relationship with a new partner, they should have their own place, and not rush in to moving in together.
If he loves you, he will give emotional support without having to be with him in a relationship straight away.
Do you have any female RL friends you can talk to about this? You might be surprised what support you can get from people who are not much more than neighbours or mums at the school gate.
Could you 'phone women's Aid, or even shelter, or the council, to see what your rights are re housing?