As the victim of an affair, what I would say is, talk to your husband. Try and sort out your marriage...and yourself. Get counseling.
One of the hardest things about my H having an affair was having choices about my life taken away from me. Had I known what he was thinking, we could have tackled our problems. Or we could have split and left my husband free to get together with someone else.
His ego was boosted by someone else. She told him he was handsome and lovely. She gave him affection and compliments. He never saw her clean toilets though, or wrestle two kids into bed or had to argue with her about who’s turn it was to scoop the cat litter. It was uncomplicated and gratifying and not real life. It’s easy to be this perfect girlfriend under these circumstances.
What is it you want? The security of marriage and a bit on the side? To leave your husband? You are being horrendously unfair to this lovely man. If you need more than he can give, then get out. If you think you’d like to stay married, get help. Be very wary of exaggerating your husband’s bad points to make an affair justifiable. If you have a problem with him...deal with it with him. At least that is honest.
Affairs take up a lot of emotional energy. My kids started to be affected by their father’s affair before I even discovered what he was up to. Almost a year later, their relationship with this father has not completely recovered. Be very, very careful. It is very doubtful that he is ego he seems. Even if he is...you are not single. The fact that he is tangling with a married woman says a lot about him. You are not star-crossed lovers, or meant to be etc etc. You are selfish and destructive and entitled. It can only end badly, if only for your husband and child.