Late to this thread, but I agree with ChippingIn. You need to put your fears of the consequences to one side almost and see the simplicity of the decision.
What you are telling us is that the behaviours that caused your H to have an affair in the first place are still very much in evidence. He resented not being the centre of your attention, had an affair and ultimately, chose the OW. He then got tired of not being the centre of attention with her as well, sacked her and returned to you.
It is a fundamental rule of affair recovery that contact with the affair partner is severed and any contact thereafter, even unsolicited, is disclosed. This restores trust after it has vanished.
That he didn't volunteer this and has only disclosed it when pressed, suggests that he will only tell the truth about anything when he is asked the right question. He possibly deludes himself that he will no longer lie to your face and sees this as progress, but it is not enough. The truth and fidelity in your relationship cannot be governed by your instincts and ability to ask the right questions.
Staying with someone for reasons other than love is never a good idea. Likewise, he sounds like a man who cannot be on his own and will make decisions about his romantic relationships based on his own comfort factors and attention needs, rather than the love he feels and is willing to give.
It is absurd that the OW was contacting him for financial advice and far more likely that she has been kept on the back-burner for some time with occasional contact, because he is probably someone who likes the idea of a safety net if things didn't work out with you. This has meant he has never fully re-committed to your marriage and you have been living with the effects of that, with a concomitant erosion of your feelings.
I suspect in your shoes I would conclude that after almost three years of turmoil, your H is never going to "get this" and I suspect it is because he never has, that your feelings for him have gradually diminished. This was always the risk he took, by not understanding his behaviour and actions.