I think you're being loyal, which is widely classed as a virtue, but ...
You're making things a bit uncomfy for your dh and
Your two friends are both adults.
I think that sometimes you just have to take a step back.
Two things occur to me: Firstly, most people do not enter into relationships with others intending to cause the other person harm. We tend to embark upon relationships with love and a wild degree of hope. That is, we love the other person and we hope, often believe, they love us. And we hope that our relationship will bring their love to us and allow us to give our love to them. I think all of us have a sense that there is something quite amazing about love; it is the closest most of us will come to a supernatural experience - taking us out of ourselves, giving us a taste of joy and eternity.
I'm banging on about this because ... crazy though all that is, it needs to be remembered when things don't work out. You can forget that it was not the initial intention of either of them to end up at this point, with their broken relationship, with at least one of them, really unhappy. In a way, we tend to forget that both of them went into this hopefully. And we "punish" as though the "baddie" intended hurt from the start. They almost certainly didn't.
Which brings me to the second point; you really don't know what was going on in their relationship. Since most people don't embark on a relationship intending to devastate another person, and most of us become involved with equal dreams of an eternity of love, something has happened to change that. You, from outside, really have no idea what that might have been. And there are probably two people who are equally sad that their hopes have failed.
For all you know, your dh's (male) friend had realised this partnership was not working and was staying out of a great deal of loyalty, although it was killing him inside. Perhaps he was desperate to have dc, didn't love her, but felt it would be an act of betrayal to leave? And then, finally, he did? That's one possible story. There are many, many others. you've come up with one story - but you're almost certainly simplifying a complex narrative.
I don't know, obviously, but really, nor do you. People are complex. I write all this having been in the position of taking sides during a divorce, in which there seemed to be a clear "baddie", only to have come to the realisation, some time later, that the "goodie" was actually completely impossible to live with, and that the "baddie" must have been desperately committed to making the marriage work to have stayed as long as they did. That said, I personally find the "impossible to live with" person rather lovely in many ways.
Although I think it is OK to judge, in fact, I think we are required to, I also think it should be tempered by an awareness of human frailty. Most of us try to be good. Some of us manage to be extremely good. Some of us manage to live within the parameters of socially acceptable goodness. But a lot of us don't, or don't always. It doesn't make us wholly bad. How awful if we are wholly abandoned just because we fall short once or twice. Seriously, are you going to, wholly, always, write him off and make your dh uncomfortable for this? You've made it amply clear you disapproved of it. Perhaps that's enough?
Anyway, four years is quite a long time. Just because you allow the man back into your lives, it doesn't mean you need lessen your support for his ex.
Sorry it's a long post, it's just that I sometimes wonder about the level of emotion on mn whenever affairs are mentioned.