How do I stop letting the other men continually ruin my life? I can almost hear one of them laughing about this
But he's not laughing. You need to take responsibility for yourself, and understand that you are creating that laugh.
Your new man can do whatever he wants. He might turn out nasty, he might turn out messy, he might turn out lovely. We all take this risk, every time we get into a new relationship. And that's fine. That's universal. What you need to learn to do is to have your own back. So, if he's lovely for 6 months, and then starts to show signs of turning crap, you need to have your own back. Currently, your exes are watching over you and judging you. You need to be the person who does that. You may still hear the laughing, but you need to understand that it isn't powerful. What's powerful is that you are moving your life forward in the way that you want, and that you will stay with a man for as long as you want to, which is for the time when he treats you in a way that feels good to you.
If you know that when a relationship turns bad, you can walk away and be fine, you won't care about what your exes would have said: you are giving them too much agency, and you fear this relationship because you give your new man too much agency.
In short, you're in charge. He gets to stay with you only if you enjoy it. And if you stop enjoying it, you move on with your head up.
What would happen if he started to show you a less pleasant side to himself? How would you feel? How would you respond?