Op, yes, first off, really, REALLY stop the affair. It is not going you any good. Actually I think you are using him. Your fantasy escape, you feel so happy and alive, etc, and how much easier it would all be if he could rescue you, magic you into a happy relationship and life... (it's probably when he realised that that he ended things).
So, once you have gained the discipline and self control that will restore some dignity and self esteem, you can focus on the actual issue.
Does your DH purport to be happy with his life? To love you?
Have you told him that you are unhappy and that you want more partnership? I have no doubt that it is hard being a SAHD to 3 children, but it must be pretty hard, too, to feel that you are out at work all day, all week, and that when you get home he does not want to share any family time with you. So what is it that he wants from you? Just your salary?
Would couples counselling help, to talk this through?
Is there any chance, given your financisl constraints, that he would consider returning to work? Childcare is so much cheaper now the youngest is 3. He could add to the household income, get a different perspective, and maybe you could both then share more family time, because for him it wouldn't be just what he does all day.
If he asks about the affair, don't lie. Tell the truth, don't make excuses, don't blame him, just apologise and keep apologising.
Take a pro-active, forward looking approach to finding out what is going on in your marriage, and how you can fix it. Put generosity and empathy into the task.
And then, if it can't be made to work, you can leave knowing you tried. Really tried. You owe that to yourself and your DH, and the certainty would itself build a more positive future.