I think you have to decide where you are in terms of your own relationship and whether you do something now
a) you confront him, he confesses all, and you can move on, great
b) you confront him, he denies all, and you wonder for ever what would have happened
Or, you wait until it has "happened" and
a)you confront him with evidence (if you think he would laugh it off as a happy coincidence)
The thing is, at what point does it become a dealbreaker?
The deceit
The shattering of trust
The "adultery"
Do you want to carry on with the relationship, or are you almost hoping that he will do it and you will have a legitimate reason to end that relationship ?