OP - you do understand that your affair hasn't just made you more distant at home, it's stopped you finding your home life and relationship so bad that you've done something about improving your life! If you had no OM, then it would just be you and your DH. That would be the only relationship you had, and it would force you to focus on it and probably would have pushed you to end it or try to improve it, but you wouldn't be in this situation you are now.
As long as you are with OM, your home life will not improve. Being with OM is actively stopping you sorting that out. Sorting will be shit and hard, but once it's done, you'll have the chance of happiness, if you don't sort it, then this is as good as it gets, a shit homelife with the occasional high from stolen time with OM. After 2 years, if OM wanted more than the occasional stolen moment, he'd be pushing you to both leave, he's not, he doesn't want that.
This is as good as it gets unless you take action. Telling OM's DW will be the one form of action that would remove control of the situation from you. The second the words are out, that's it, you are no longer in control. You will have to just have to then sit back and let everyone else move around you and at best, are just in damage control - and trying to do damage control for your DH, your DCs, your financial situation, and probably without support of extended family and friends. You will be forced to end your marriage at the point your DH finds out. It will be on your DH's terms. Not yours.
End both relationships. I would put money on your OM panicking if you said you were leaving your DH and actively trying to stop you, encourage you to wait a few years. Your OM isn't going to be a help to your long term mental health.