A friend of my mothers started off like you. She was divorced, "fell" into a relationship with a man who was married with a son. He claimed he couldn't leave until his son was older as it would destroy the family. And so it went on. For years. And years. She has now been his mistress for over 30 years. His son is grown up and he has grandchildren. They are in their 60s and he still is with his wife and seeing her on the side.
She has settled for a part time relationship with a liar and a cheat. She never had her own children. She still lives alone in her flat. She tries to fill her spare time now she is retired with hobbies and friends - most of whom are couples who feel sorry for her.
She sees him twice a week and they snatch the occasional weekend away. What a waste of life.
But the worst thing was when he got sick.
She had no idea what happened. He just stopped answering his phone. Because you see, the people who were called by the hospital were his REAL family - his wife, son and grandchildren. She had no idea what had happened to him, she couldn't go to him in the hospital because his wife was by his bedside. She eventually found out through a mutual friend and got infrequent updates on his state of health.
He did recover. But now she knows what will happen when he dies. She will be waiting for news like any other acquaintance. The person who will be by his bed, the person who will be sitting in the funeral car, the person who will be comforted by his loved ones will be his WIFE. Not her. Never her.
She will be alone, unremarked, unremebered, just a dirty, shameful secret that, please God, his family never discovers.
How sad. Perhaps this is what waits for you.