OP, I don't even know you, but when I read this, I felt SO VERY angry on your behalf. He is taking the total PISS. He wants to go on a double date with his Ex-mistress, whilst you sit at home.... and he thinks you should be okay with that? Is he fucking insane? He actually sounds like a narcissist. No one could be that thick.
I know how hard it is, but believe me, you have to leave him now. This just isn't going to get any better. My first H had several dodgy encounters, with several OW. It took me 4 years to leave him (we had shared children and assets), it's very hard, but....it was the right thing to do. You can't live like this. It destroys everything. I actually had a breakdown of sorts. I now have a lovely DH, who is so much more loyal and trustworthy.
You're only 35. Start making an exit plan now, you are still young enough to start over. And above all TAKE BACK CONTROL
Once I had started the process of working out the money, looking at property, --and flirting with other men on nights out the whole thing about starting over, became rather exciting.
Funnily enough, once ExH knew I was leaving him, I was suddenly the only woman he wanted. Funny that.
If you have the means, such as parents nearby, I would pack a case and leave when he is collecting the garden stuff on Saturday. Let him come home to an empty house and let him feel the full force of his actions. Don't answer any texts or calls, and don't tell him where you are. Chances are, he will think you've popped to the shops, and the realisation that you've actually had the balls to leave him, will hit home around dinner time, when you don't come home.
Anything less and he's simply not going to get it, and (in the nicest way possible), you would be a doormat.