"His argument is that if he wanted to be with her he would have ended things with me (he can afford the divorce so it’s not about money)."
@Violetrose1
Bullshit. He doesn't want to lose his 'guaranteed home comforts' and his 'upstanding family man' reputation. Right now he gets a clean house, cooked meals, an 'on call' companion for TV nights & outings, and (I assume) a sex partner. You are the 'known', OW is the 'unknown'. And also, people in his orbit probably think he's a 'great guy'.
Personally, I don't think the 'affair' ever ended. He's keeping her 'in reserve' against the chance that you'll tell him to GTF. Which you should. Oh sure, perhaps he stopped sleeping with her (or maybe not), but I believe the 'emotional affair' has continued throughout. He's stringing both of you along. And I think his 'we're just friends' thing is to pave the way for him to either resume the physical affair, or make it easier to continue it if it's been ongoing.
A good friend went through the same thing. He got caught, didn't want to lose his home comforts and 'family man' reputation, so he (supposedly) broke it off with OW, went to counseling and 'said all the right things' to make my friend believe he was truly remorseful. Only for my friend to discover that he'd been lying the whole time and had never broken it off. She kicked him out, divorced him, kept the house (now paid off) and has lived happily ever after. He married OW, they bicker constantly, and are in heavy debt with a huge mortgage.
See a solicitor before you do or say anything. It may seem easier to you to walk away with only what you walked in with, but it may not be in your own best interests in the long run. Don't make an uninformed decision. I'm not telling you to 'take the house' or 'take him to the cleaners' or even that it's possible. But you do need to know what your rights are WRT marital assets and if claiming a share will make it easier for you to set yourself up in your new life.