Yes, I do think your own personal experience is colouring your judgement here.
As many on this thread have recommended, I think you should stay out of this.
First, you do not know your neighbours and have absolutely no responsibility (or business) interfering in their personal lives, no matter how well-intentioned your intervention may be.
Second, you do not know the that you’re right about this — perhaps he was seeing a sibling, or a close family member.
Third, you do not know their circumstances — some people live in open relationships; others know their partners have affairs but prefer to keep it to themselves in order to preserve their families. Outing the husband and the affair might put pressure on the wife to leave him when she might have preferred to turn a blind eye for the sake of her children, for financial reasons, or whatever reasons.
Finally, loss of children & break-up of families can have a detrimental effect on people. What if you do out him and he holds a grudge on you? For your own safety and peace of mind, stay out of it.