Okay, I've just read the entire thread.
I'm not going to comment on the porn use as I don't think it's relevant and it's more of a side issue.
The main problem here is that your partner had a ONS which left you feeling crappy and smashed your self-esteem to bits. You said yourself that you just 'muddled' through the aftermath, which probably means you had blazing rows, made up eventually and forgave him. I don't think you ever dealt with the issue properly which has led both of you to where you are now.
You resent him because he had sex with another woman. You probably don't know why he did that and if he is going to do it again. It's probably sent you into paranoid city about what he's doing, who he's with and if you can ever be secure in your relationship again.
You've said you don't reject sex but he is probably actutely aware that he royally screwed up and doesn't know how to act around you. I'm not saying that he's being fair or reasonable in what he's doing, just that he's probably feeling quite uncertain too.
My advice would be to really sit and talk about the ONS. Why he did that, his reasons, etc. (Men in happy, fulfilling relationships generally don't go shagging around). You need to be completely honest with each other and really try and work out the roots of the problems your having.
A man who really cares for you and about the relationship will want to work on your issues to try to fix them. He won't yell at you or make you feel bad for bringing it up as he will understand that it needs sorting out.
If he says no or yells at you then kick him to the curb (easier said than done, I know). You are feeling unhappy and unfulfilled in your relationship, attempt to fix it and if that doesn't work then there is nothing left to do other than leave.
Everyone deserves to be happy.