Living in a state of uncertainty like this is awful. You need to decide and stick with it.
My opinion - which isnt worth much, we only have a tiny snapshot, you know the full picture - is that there are enough reasons to stay. But it would only work if you really gave it 100%. No more hankering after the other man, no more what if's.
I would say - and this is how I always feel about these situations - I really dont think it would be fair to take your DS straight into living with OM. He's at such a crucial age and that kind of thing is so bewildering and confusing for small children, even older children struggle.
Your relationship with OM, if thats what ou choose, should absolutely take second, third, fourth place to your sons emotional security. It'll be hard enough for him to process whats happening between mummy and daddy.