I spoke to him about how I felt and asked him if he loved her. He said that he of course loved me but there was a part of him that did love her but he didn't know in what way and he has felt like that for years but would never do anything to hurt me and our girls. When I asked him if he wanted to be with her, he said there was no point discussing that, however now I can't get it out my head
I know that many people would say that he is entitled to have friends of either sex, and yes that is true, but when it comes to your comment above, the "friendship" crosses the line into emotional affair territory.
Admitting that he loves her "but he doesn't know in what way" tells me that he probably does know in what way he loves her , but he doesn't want to admit that to you. I'd put money on the fact that there is an emotional affair going on, and probably has been going on for years , under the radar , with you and her partner none the wiser.
What you do about this, is up to you. Can you live like this now, always wondering if he is going to come home one day and announce that he wants to be with her ? If you can't live with this, you might start looking at separating , as painful as that might be.
I was in a very similar situation many years ago, and I spent a long time feeling like it was all in my head, until I came home early one day and heard them talking in our back garden. Their voices sounded like lovers talking , and in that moment I knew that my suspicions were correct. I only wish I'd figured it out sooner. I got a divorce, and it felt so freeing to be away from that constant worry and suspicion that I'd lived with for all that time.
Only you know what your next step will be, but I'd say that in your situation I'd be looking at separation to give yourself some breathing space . Good luck OP.