OP, if he won’t go for counselling then there really is no way back.
I get the finding it hard to separate when you’re still good friends. I went there with my eXH. We lived together under the same roof until I moved out after 8 months, and I can honestly say we still remained friends, and I thought we always would.
Even after we’d split, he would do things for me, I would go in for a drink when dropping off the DC, we had chats etc, and members of my family said we should probably never have split up.
But the thing which made the difference was distance. It’s harder to separate yourself from someone when you’re still in each other’s space, because you have that familiarity of being together, and even if there has been infidelity, feelings do still remain.
But over time we both moved forward. He started going out on dates, and eventually I got together with someone else. Once he was in a relationship his new partner essentially forbade him from ever having any kind of dealings with me, and obviously as time progressed that just became the norm.
For you the hardest and yet the easiest way to separate is going to be to move out, or have him move out.
The affair is a red herring at this stage. Not because it doesn’t count, but because in spite of you both loving each other, the chemistry between you both has changed. The affair is the cause of that, but now that the affair has finished, the change still remains. And if you can’t live with it, then the only solution is to walk away from it.