Trying to see this from your husbands point of view for a minute.
He's in a relationship with seriously mismatched libidos. Presumably he has spent years trying to figure out how to solve this. Eventually having exhausted all other options he suggests getting sex elsewhere.
Initially his wife agrees, and he thinks he has a solution. Not the solution he really wanted perhaps, he'd probably prefer to be having lots of sex with you, but maybe a solution that manages to keep the marriage going.
And then you decide to stop what little sex you were having. Your husband is naturally upset by this, this whole situation started because he wanted more intimacy and affection with you, and instead has ended up with no sex with you at all. But he accepts it, because he loves you and wants to stay married to you. And if you don't want to ever have sex again, then this setup seems to be the best of a bad bunch.
Except then you tell him, that actually, you might want sex after all, just not with him. So now not only is not getting what he wants, but instead a poor substitute. But some other guy might be getting what he wants instead.
You can see perhaps why his head is in the shed a bit?
I'm not saying he's the one in the right here. He should probably have accepted years ago that his love for you isn't going to be able to overcome the lack of something he desperately needs in a relationship, but that you don't want. He should have split up with you when he realised this, rather than trying to change you. But he's done this precisely because he loves you, and you've pulled the rug out from under you repeatedly.