I don't think you have to completely bite your tongue to stay neutral.
You can't go on and on about it, but I think picking your moment to say to the person who had an affair, once and directly how you feel / see it can be great for clearing the air for the future.
Not comparable, but when I was a student, a friend of mine (let's call her Sarah) stole another close friends (let's call her Clare) boyfriend. Sarah and the boyfriend slept together the night Clare and boyfriend broke up, while on a big group holiday. Clare got up in the morning to find her ex of about12 hours naked on a sofa with Sarah.
Sarah and boyfriend were together for quite a while subsequently.
As a close friend of both I was in a v tricky position. The next time I saw Sarah I said 'I need to tell you I think what you did was completely out of order.' She said 'I know'. It cleared the air and then it was easier to move on and stay out of subsequent fall out that wasn't my business. A few other mutual friends did similar. Think it helped Sarah realise how out of order it was (the boyfriend was out order too but I didn't know him nearly so well.)
I remained friend with both. Sarah learned from her mistakes, has never done similar, and remains one of my closest friends 20 years later.
I think you are entitled to say your piece, briefly, once. And doing so is likely to make it easier to move on in due course.