I had a friend who did exactly this, something over 2 years ago
She didn't have the grace to end her marriage first. She did the emotional affair, the full-on affair that broke up the marriage and messed up the kids, the subsequent guilty avoidance of the kids' father.
Without knowing the full facts, from what we have seen, the father has tried to behave like a saint, much to the detriment of his own mental health. The wife saw it the way of the OP here - inevitability, destiny, love of her life. It's become more prosaic over time: like any relationship, the one with the new woman has its ups and downs. Her mental health has suffered, and she has dumped many people who had thought she was a friend. Her kids are much more messed-up by it than she had thought they would be.
So in the end, the great love affair has turned into a prosaic, normal relationship with a lot of fallout on people who didn't necessarily have to be hurt quite so badly, and a considerable loss of support from all the dumped friendships.
A better way to do it would be to have the manners and personal strength to end the marriage properly, and then take up with this woman. Behave like an adult, don't treat your husband badly, don't lie to your kids or your husband.
At the moment, if you're thinking about sex with the colleague, you're definitely treating your husband and kids pretty badly. Have the guts to do this kindly and gently, if you must do it.