You don't have to view your past in any other light than how it really was.
My really firm advice to you is to trust your own memory (aided by your own mementoes) much more than that of someone whose brain is currently addled by chemical lust and infatuation.
We tend to think of the 'being in love' phase as a positive thing and for the person experiencing it, it is - for the most part. But even the most hardened romantic who is addicted to this state and constantly tries to recreate it will admit that it intrudes on logic, rationality and seeing life as it really is - or was in the past.
If I were presented with a life or death situation where I had to trust someone's memory of events, I'd pick the person who isn't in the throes of an intense infatuation. This is because I've seen so much disonnance between couples where one has a very sharp memory and is recalling love letters sent, weekends away, surprise parties organised and most tellingly, the everyday acts of someone who loves his/her partner - while the other is claiming that love flew out of the window at some point before the OW/OM's face appeared through the panes.
Letting go of this man doesn't rob you of your past therefore. Don't let it be contaminated by someone who has a need to re-write it to justify his current and past actions. A very good project for you in the future (it will be too painful right now) is to sit down with diaries, calendars, photos and mementoes and trust yourself. You are more likely to remember the good and the bad, because you have no need to polarise things and describe life as all good or all bad.
My sense is that your husband is still in touch with the other woman. It is the main cause of ambivalence by the infidelitous partner after an affair.
If your husband goes to her, having managed to manipulate you into being the one to end the relationship, you're right he will probably not feel an immediate sense of loss. That's not to say he won't in time, but usually by then it is too late.
But an important issue in triangles is that the relationships prop eachother up, at least for a time. The person at the top of the triangle is getting his needs met by two people and is usually experiencing a surfeit. When we have a surfeit of something, we have little motivation to choose one provider over another or to risk ceding all our needs to one person who on her own might not meet them at that point. Bear in mind too that the one at the top is being hugely selfish and thinks about his own needs first.
Only the withdrawal of one of the triangle and the loss of that person's contribution causes the person experiencing the surfeit to evaluate what's left. Since new relationships usually have a honeymoon period of up to 2 years, that evaluation might be some way off. Conversely, many affairs burn out spectacularly quickly once out of the bubble of illicitness and secrecy, when a real life of bills and hurt children intrude.
However long it takes though, you will be distanced from it for the same period and especially if you use that time profitably to heal and get back on your feet, your life will be much better than this hinterland of crushing low esteem, living with a person who doesn't love fully and feeling exhausted by your fruitless efforts to make him choose you in an active, wholehearted way.
Have courage and be strong. I have never known a woman get to happiness through the path you are choosing. I have known many who got to happiness by standing up for what they are worth and not compromising their standards.