What do you want to do? Carrying on as you are clearly isn't an option for the man you are having an affair with, and rightly so. He deserves better.
It also isn't fair on your husband. He also deserves better.
If you want to continue the relationship with the man you are having an affair with, you need to end your relationship with your husband. You don't need to tell him why - or your parents and kids for that matter. You can just leave, if the man you are having an affair with is happy for the secrecy to continue for longer. This is the coward's option and still leaves you open to judgement from your parents and children. However, that's no reason to stay in an unhappy relationship.
If you don't want to continue the affair or stay with your husband, you need to end the affair and your marriage, and manage the fallout with your parents and children.
If you want to avoid any conflict with your parents and children, you need to end the affair and go to marriage counselling with your husband in order to get to a place where the rest of your life with him is one that you want.
I don't really understand why you're asking what to do.
If you're asking what you SHOULD do, that's entirely up to you and what you want from the next chapter of your life.