Short answer - yes, you should be worried. This is either an emotional affair or heading rapidly into that territory.
My DH put me through this for the best part of 2 years and it is only after a lot of soul searching, therapy and cutting complete contact with OW we are now back on track. But our marriage has fundamentally shifted and whilst he now does everything he can to reassure me I can trust him, is open with devices etc I will always hold a tiny bit of resentment, which I try to never show but occasionally rears its head.
He said all the same things to me ‘she’s just a work friend’, ‘I can be friends with a woman without feelings involved’ ‘you’re controlling me by telling me who I can talk to’. I drove myself crazy thinking I was an unsupportive wife for at least 12 months. He was constantly messaging her (she’s very attractive of course), during our time together in the evening, when we were out together, at night in bed together, first thing in the morning. He showed me a few messages at first, then start deleting them, then moved to Snapchat. He swore he wasn’t attracted to her. He even promised to reduce contact a few times but it never lasted long. The thing I really couldn’t forgive, apart from the intrusion into our life together, was he started treating me differently - at a work do where she was there he ignored me and they were giggling like schoolchildren together.
I became a shell of myself, crying all the time. He would get angry and defensive if I asked him to reduce contact, he even said I was ‘holding a gun to his head.’
It took me a long time to realise it would never end. I also managed to get into his email and found some overly flirtatious emails, which he said he’d never send again.
I eventually told him (and meant it) that he has a right to ‘have friends’ and not feel controlled and do what he wants. I also have a right to do what I want - and I don’t want to feel second best, ignored, emotionally unsafe and unhappy. Even if (a big if) you are just friends, it is making me unhappy and my choice is to exit this marriage in the next 5 minutes if you don’t end it with her. He cried and messaged her and cut contact there and then. We then had individual and joint counselling to understand how/why this happened and recommit to our marriage.
Like I said, things are much better but I’m sometimes taken back to the sadness and anger that my best friend could put me through merry hell. He understands this now but says at the time he really couldn’t see beyond his own needs to how much he was hurting me.
I don’t know if this helps, but please know you are not losing your mind. Trust your gut.