Gonna, there is so much wrong here it is difficult to know where to start.
I'll start with your H though. He knows that you don't believe he loves you, finds you attractive, or that he wants you, not OW. He knows he's never told you why he treated you so badly and why he had an affair. He knows you're hurting and that you need help. He knows that pulling his weight domestically and putting you first would help enormously. He knows that showing genuine sorrow and contrition would help more than anything. He knows that getting some therapy of his own would help him to face up to his selfishness.
Despite all of this knowledge, he does nothing to give you what you need to help you.
You know that the root of your problems is low self-esteem, little self-respect, a passive-aggressive way of dealing with problems, a lack of assertiveness and an expectation that your happiness will be defined by your marriage and not other life enhancers such as friends, absorbing hobbies, work that you adore.
You seem able to convince yourself that you can put up with him with all his many flaws and that you won't end the relationship. That's a shame, but if you really cannot let go, then you've got to find a way to live with that choice.
So it seems that you've got to work on yourself and build a life which doesn't revolve around him and the boys. My self-esteem is defined (and always was) by things other than my marriage. My work, my professional reputation, my mothering, my friendship skills. I derive joy from all sorts of things and not just my marriage and my family. My H knows all this and it is why he respects me.
I don't mother him - our view is that building a home and a family is a shared enterprise and we all help eachother out. If I'd been invited out somewhere like you were last night, it wouldn't even occur to me to cancel, especially if I didn't go out often. I would have said I was going and asked him to stay in, or sort a babysitter. Childcare is not my responsibility, it is shared.
However he cannot read your mind. You didn't tell him about your invitation and didn't tell him that you needed him to be in last night. You didn't tell him that you needed him to clear that washing now. You didn't ask him to do the shopping. When he didn't read your mind and failed to pick up on your sighs and no doubt huffing and puffing, you exploded. This is classic passive aggressive behaviour and it is infuriating to live with.
From your H's point of view, what does he see? He sees a woman who makes his life comfortable and he no doubt thinks you are a great mother. But does he respect you? He sees you acting like a doormat towards this friend who takes but rarely gives and sees you forgiving that same friend (and him) for behaving inappropriately with one another. It sounds like this woman haunts your home most evenings.
He sees a woman who constantly needs to be asked to go out and have some fun and who puts her needs last every time.
Putting yourself first occasionally and building a separate life that brings you joy actually builds respect in others.
It will take a lot of practice, but if you worked on expressing your wants and needs in direct and clear terms, it would help build this respect. Learn to say "No, that's not acceptable" or "No, I am going to do this" or "No, this is what I want".
You seem to live a life full of dropped hints, suggestions rather than requests and you seem to expect him to read your mind. You in turn try to read his, rather than asking him what he thinks or feels.