I'm infuriated at the situation he's putting you in.
He's trying desperately to have an affair and failing because he's too flipping old to be attractive to this woman. (The failed pity fuck attempt is just hilarious).
But he wants you to meet up every few days to discuss the situation and rub your nose in it. Telling you where you've gone wrong. Oh - he doesn't like doing the crossword with you, and oh - you only work part time. (Perhaps you would work full time if you were not caring for someone with serious mental health problems!)
He's got both you & this OW dancing round him & he's revelling in the attention & the power. Chump Lady is absolutely right that your husband wants this competition to go on indefinitely. It's all food to his ego.
The implication of all this - utterly false and utterly disrespectful to you - is the following:
the OW is better, she is who he wants, but he can't get her and he'll hang out with you for pity because you're so upset about him leaving.
He will tell you where you went wrong (crosswords, holidays etc)
All the while underlining that your role in his life is to be his carer: it's your job to make sure he's happy & healthy, any bits you miss he's entitled to get from another woman.
It's all complete nonsense of course. Disgraceful. You are clearly such a lovely person, and if anyone does not deserve anyone here, he does not deserve you.
The only way to make this work is to pull the rug from under him completely. Ditch him (or make him think you have) and become the unattainable woman, the one he had, but lost. The one he aspires to be with, but can't.
At the moment you're chasing him & he's loving it. As long as you do that he will identify you as the one who's not good enough.
Once he sees he's lost you completely, his so-called feelings for the OW will collapse. He'll become critical of her. She'll become inferior because he has her and not you.
I know how much you love him, how much you want to see him and understand what happened. But if you continue seeing him he will never face reality. I don't think anything he's saying is real anyway, so it's not even helpful.