He sounds a bit nuts, to be honest. But there are some contradictions - for instance, he cannot forgive you for an infidelity before you were married. If, as he claims, he does not love you, then why does it matter whether you were unfaithful anyway? Or does it mean, as I suspect, that he sees this as evidence that you do not really love him, and has decided to spend his life beating you over the head with it?
It sounds as though he is treating the marriage rather like one of those arranged marriages where you hope that, in time, you might grow to like each other. Honestly - just as if you were living in a country where you had no choice but to have an arranged marriage.
Even where a marriage is not arranged, people generally enter into it with some sort of bargaining going on (eg: little thought bubbles about "well he is not the best looking but he will be a great family man or "her family are a bit weird but I like her more than anyone else" etc, etc.
This is fine, as long as the expectations are reasonable and you are both reasonably honest with each other (don't think people are ever necessarily 100% honest, even with themselves.)
However, it sounds as though he was completely dishonest with you in that he pretended that he loved you, when in fact he didn't. Or - he has decided that he wants to use that infidelity as an example of how you don't really love him.
God, he just sounds nuts to be honest. Unless you are willing to be in a marriage of convenience? Having said that, I do seem to meet an awful lot of married people who are quite horrible to each other, and I suspect they are only together because of inertia/children/shared finances/what the mother in law might think etc, etc.
Sorry, probably not helpful. However I suspect you are most definitely not alone in living in a relatively loveless marriage, not that that is much consolation. But why does your partner feel the need to throw this in your face - that is what bothers me. I would say that is verging on verbal abuse. For instance, you would not accept it from a friend.
If your partner is treating you worse than a friend - that is showing complete disrespect, then surely there is something deeply wrong. In fact, he is showing total disrespect for your feelings.
Counselling??