The more I read, OP, the more sympathy I feel for you and the more I think that it's the 'jumping off me' mid-sex that's the really weird bit. Therefore, understanding that is likely to be the key to understanding his behaviour, including the headline problem, which is the slowness and lack of stimulation for you.
I think it's reasonable to see the 'jumping off you' as part of the whole 'stop-start' syndrome. If your sex session was plotted on a graph, with intensity against time, you'd see a variable line, never getting very high, and alternating between slow and very slow. And then sometimes dipping to zero. So maybe the 'jumping off' is the 'dead stop' position on his dial. And, sorry, how many men stop, pull out and literally walk away from their partner during sex, before they've come? It's practically unheard of.
So something is bothering him, either physically or psychically, during sex, and he's not being honest with you. If you're sure it's not physical - eg tight foreskin/cramp/getting tired - then it's something in his mind. Either way, it's crap that he won't try to explore this with you.
The word 'nasty' keeps coming up in relation to what you want. You say what you want, he says you're nasty for wanting that. So does it come down to this? - in his preferred version of reality, you're not supposed to want anything at all, sexually speaking. It's your job to just lie there, maybe thinking of England, but definitely being passive, while he 'pleasures' you (but ofc doesn't). It's you having any sexual desires at all that's 'nasty', certainly any desires that you define yourself, as distinct from what he serves up.
I'm inclined to wonder whether he has shame issues about sex. The raunchier it is - certainly on your part, but by the sound of it on his too - the more shameful he feels it is, hence his insistence on this attenuated, low-intensity version, so un-raunchy it's almost not sex at all - and of course when he jumps off you, definitely not sex at all. I'd also be willing to bet that he gets so little stimulation from this terrible sex, he begins to lose his erection - and these are the moments he jumps off you and starts making excuses. Did he really go and meet his dad, or was that just an excuse?
And of course, you having an affair was - in his mind - shamefully abandoned. You went off and satisfied your nasty desire to have nasty, hard sex with a nasty, hard man, and you dared to enjoy it, and then tell him you enjoyed it. He was provoked into 'punishing' you once with nasty hard sex, but then was ashamed and resolved never to be sucked into such raunchy awfulness ever again.
So perhaps he's trying to make a passive woman of you, @Creatingsacredspace. Maybe there's a pedestal under construction also, for you to stand on, constant, unthreatening, not moving, certainly not thrusting. Because that's the only version of you he can handle? He wouldn't be the first man to be afraid of women with sexual desire; they used to call them witches.