As for being driven to it...
Funny how he wasn't driven to talking to you about it, insisting on counselling, isn't it?
My XH fucked prostitutes. For a long time, I had no proof. He only looked - was I going to tear up my newborn child's family over that? I couldn't have sex with him though. Made my flesh crawl. He never hassled me - because of course he was getting all that he wanted / paid for.
My point is - I went several years without sex and I was, frankly, gagging for it!!!! Despite that... I never cheated. Don't think many people could be more morally justifiably "driven to it" than me. Still didn't.
Not because I'm some paragon of virtue or unusual moral steel. Because I'm normal. Because actually, no matter how much you are gagging for it - nobody ever died from not having sex.
He's an arsehole, and you know it - but as you've chosen to accept it, it will now leak into your soul in a thousand other ways. Good luck - it's a shit life, and you won't realise just how unhappy you are until you stop accepting it* and feel the weigh lift.
*obviously I think that's through dumping him, but it could be through counselling where he really comes to understand that he has treated you like shit and blamed you for HIS arseholian betrayal