Of course you are NBU. How very hurtful.
Hes playing games and having a secret life with this other woman, either in his head or maybe even meeting her too. It is never acceptable to have secret friends in a marriage. He has betrayed you.
Take the phone off him and tell him to leave your home immediately.
Give yourself some time to decide what you want the outcome to be from this.
How has your marriage been recently? Do you consider yourself happy or are there other signs that is going wrong?