Hi OP. I'm sorry you are facing this. FWIW this is my view -
I believe that people should try and give their best to their partner, not others, and if others get in the way, end that association or at least curtail it to make the partner comfortable.
He is saying he won't do that, he won't put you first. Therefore I would tell him that his behaviour is no longer good enough for your relationship standard and as such, the relationship is over. It is not controlling in your part to feel threatened by his feelings for another woman; it is normal. He is your person, not hers.
Someone else said, then ask him to leave. You will see where he goes. He can do one of two things - go somewhere, think it over, realise what he has risked then come back and change that work friendship. OR he would think, 'right, I am free now' and go to her. You will then know what his true feelings were, and so will he.
Once he is with her, facing divorce, everyone knowing etc, he might realise what he has done and it all goes wrong for him, or he could stay with her and change the narrative to 'my marriage was good but it has come to a natural end.' In this case, you will be glad you bit the bullet and it will be better for you for it not being drawn out.
It is a terrible shame tho and you must feel so sad and disappointed. Either way he is being unkind and mean and his behaviour IS detrimental to your marriage, whether he accepts that or not. I would call it, right now, and give him his head and see where he goes.