OP,
I'm married nearly 30 years an I think I know a bit about love in a marriage.
Marriages at times can be trying for a variety of reasons, often from exhaustion with children, money worries, health worries, employment worries, but it is your genuine love for each other helps you both muddle through.
But it is NOT love to treat your partner like shit for years, it's just not.
Whatever he feels for you, it is NOT love.
Love is so not being nasty and unpleasant to your partner for years.
Whatever he is grieving, he is not grieving a real love.
Whatever he felt for you, it certainly wasn't real love.
You can't treat a person you truly love, the way he treated you for years.
I hope you can understand that, and really get it.
The reason your marriage failed is because HE NEVER loved you.
Your affair IMO doesn't figure in the ending, simply because your marriage has been dead a long time, because HE never loved you.