Listen MugShot, your question was Am I being taken for a complete fool.
As loathed as I am to allow anyone here to call themselves a fool, I will have to concede that you are not acting in your own best interests or in the interests of your supposed relationship long term.
Now I don't know what came first, the Abuse or the Affair. I'll place a bet on the ABUSE tbh, but perhaps it was so low grade you didn't realise it. It's OK, that's what always happens. If these 'men' were upfront about who they really are, they'd never get a date let alone a look in at a normal life.
this man has a sense of entitlement that is unjustified. He thinks he's entitled to call the shots in your life, to treat you as he sees fit, to manhandle you and to betray you. He has no thoughts about you, your well-being, happiness etc. I doubt really you figure much on his radar, bar in the capacity of W.I.F.E.
(Washing, Ironing, Fucking Etc). I'm sorry, I know that hurts, but the sooner you realise it, the sooner you regain control of your life and your happiness. Right now you need to understand that this man is the ENEMY of your soul, he'll stop at nothing to bend you to HIS will, and your feelings don't come into it. It's a straight fight between HIM and YOU and he won't play fair in his objective to crush you.
The words that follow will be hard to take on board, they may prove difficult to execute, but it is truly what I recommend for you to recover from this as soon as possible.
One fact before we start: Abusers don't change. Oh there may be some belligerent liars posters that come behind me and say their men have changed, they have seen the light and they don't abuse any more, but tbh, these 'men' have merely adapted their behaviour, adopted the Dr Jekyll that's used to reel in their victims. They know they'd be out on their hairy arses if they start trying to rule the roost again, and they know that they are so inferior as human beings, no-one would have them. Their partners have papered over the cracks and are sticking their fingers in there ears and eyes and saying LALALALALA to drown out any hint of a residual abuse problem. In the exceptionally rare cases of men reforming, it is when the entire family, all his friends, acquaintances and everyone he's ever known desert him due to the abuse, totally isolate him and cut him out of their lives. How often does that happen?
So on this basis alone, you have to be the bravest you ever have been and tell him to go. You have to do whatever it takes to get him out of your life.
Have no sympathy. He had none when he kicked in the door, when he dragged you bodily downstairs, when he yelled at you. etc etc etc Be so cold you give Absolute Zero a run for it's money.
This will hurt your heart more than anything you have ever felt, I know it hurt me, my throat ached with a million un-cried tears for a month before X left, but I had to let him go, I knew that it was a foul relationship, that had to end. I knew he hated my happiness, that he was jealous of his own son's happiness too and that in time if he couldn't get our DS to gang up on me, he'd turn to crushing him too.
Now the AFFAIR - This is another reason you have to end it. without the abuse, I'd say there is a reason to fight for the marriage, but it is a fight that both need to undertake, on their own terms. He's not fighting for you, he's hoping you'll STFU and drop this.
For that reason he needs to be slung out, he needs to be told to go. Any less and he's got away with it.
Now adding the 2 issues together, you need to tell him to go due to the affair, but you need to KEEP HIM GONE due to the abuse.
There really IS no other option on the table for you. You can't stay with him, not after this. Do you see that?
Life without this man in your life can not be any worse than life with him. Trust me.