Can we just address the sweeping generalisations repeatedly saying that kids who are from divorced parents have "ruined lives?"
My children have two parents who love them fiercely, we don't see eye to eye, but there's stability, structure, two happier parents apart, double Christmases and birthdays, two extended family events, support and love. My kids are thriving in school, do extra curricular activities, we have holidays. I work part time in a professional job. I bought my own house from the split of family home in a less nice area but we love it. Its ours. And the peace of that feeling is UNREAL.
Assuming some financial independence or ability to obtain it OP, your kids can absolutely thrive with separated parents. Do some reading online, tons of insight from psychology experts.
It's not the ideal set up of course, but for other women to suggest you repeatedly have sex, drunk, with someone not of your own sexual orientation because you're married to him is downright disgusting, and to keep the peace in everyone else's lives. Like you don't matter?
You'll become a shell, and you deserve more than that.
If you're looking for permission, and not being regarded as selfish to protect your own mental health, then you absolutely have it.