no one should be forced to live in an unhappy marriage, we only live once.
That depends. How hard and how long did he try to make it a happy marriage? Having babies and toddlers can be a very tough stage for any marriage. He chose to have children with her and then left her not long afterwards. So was she showing signs of serious mental instability before he decided to start a family? If she was, it seems like an odd decision. And if not, then why did he leave so soon? Either way you are right, he left a mentally vulnerable woman alone caring for her very young children.
From what he said, he had a couple weekends away (work trips + staying for a couple of days after for sightseeing / travel), and she went crazy suspecting he is cheating, going through his phone and email, accusing him of imaginary liaisons with work colleagues etc.
Well, he is starting a liaison with you, and you work with him. So her version is not impossible. And he's warning you that if you do have a long-term relationship then you will be in a tricky position. If you start to wonder what he's up to there will be no reassurance from him. You'll just be the next irrational crazy ex.
And when you do a lot of travel already, and you have young children and a struggling spouse, extending business trips for sightseeing is very selfish.
I don't know why he does not have them overnight, he travels a lot for work maybe that's why.
No, that wont be why. Either he will be able to fit them into his irregular schedule, or he'll adapt his work practices to fit his children's needs.
There may be a good reason. But balance of probabilities says all this is a bit dodgy so proceed with caution, if at all.