Why are you so sure your family will be unhappy if you (accept your feelings are valid and) leave?
Financially, it becomes harder to maintain the family home and perhaps it needs to be sold and there will be upheaval and the visions for what different people had of their nuclear family unit will need to change.
This doesn’t mean greater unhappiness also results. It might be the total opposite. And yes there might disapproval all round - to begin with. But your children are adults and you surely have waited long enough?
IF.. you leave and become happier and more fulfilled and settled, and your now husband has a chance to start over, and gets the chance to have a full physical love life and a fully committed partner too, isn’t that as /more important than what others think or say?
Put this way, if your daughter (the one with the child) was in the position you are in now and at the exact age and stage of life you are at, what would you like her to know?
“Stay”
Hold the family unit together at all costs and deny all of your own feelings and wants…
“Leave”
Your needs and wishes are as valid as everyone else’s, you have already ignored them for too long, and they still haven’t changed…
Something else?
I agree with other replies saying that if you leave you must do it for yourself, not this other man. And that building a better life for yourself should be your number one priority, not fitting into a new caregiving role too fast.