OP, you say it is not a big deal. The thing is, it's not a big deal for you, but it might be for your husband. This is about HIM and his reaction, not how you feel about it.
Tbh, the fact that he is a close, dear friend, might make it all worse, from husbands point of view. I'm sure he will understand it was all years ago and only a few times and dint mean anything.....but that doesn't necessarily reduce its significance in someone's mind. Feelings about this kind of thing are by definition not logical. And people react differently, so the fact you or I might feel one way, does not mean husband will or should feel another way.
You sound to me, as if you are having trouble prioritising our husband over this old friend. Yes, he is an old, close, dear friend.....but husband is on a different league altogether and deserves priority. Cancellation maybe necessary and you really need to be prepared to do it, if husband wants you to. Yes, will be difficult with old friend, yes, might create difficulties for him with his wife.....but these are the least of your concerns or should be. Relationship with husband should be top priority. Don't be willing to put him in a position where he feels he cannot say he would rather this other bloke doesn't stay, because you don't want to have a tricky conversation. If he feels like this before the ex comes to visit and then has to accommodate him, who knows what this will do to your relationship into the longer term. Forget that it's not a big deal to you, or to ex, or even to exs wife, think instead about your husband.
Seems to be consensus that he should be told, and that you and ex knowing, whilst husband doesn't is a bit like having a 'dirty little secret' which would be humiliating for husband, if it came out whilst ex was with you. So glad you're not willing to risk that.....hope it's not just because you would have been caught out, but because of genuine concern for how husband would feel.
Tbh, fact that ex is close friend, is an issue I think. if he's a close friend that you talk to often etc, it will seem odd to husband that you never mentioned previous nature of your relationship before. I don't think there is an issue of concern about hi. Not trusting you now or thinking you and ex are likely to have something together now, it's just easier to emotionally deal with the idea of your spouse having had sex with people in the past that now mean nothing to them, rather than with someone they are still close to.