Well I decided to have a chat with him last night as firstly I hate playing games and secondly, as much as I appreciate all the advice given on here I feel like this thread had started to take on a life of its own and was freaking me out possibly unnecessarily.
Firstly I was honest with him about the fact that I hadn't enjoyed the threesomes and swinging as much as I thought I would and that I never wanted to do it again. He said ok and that it was always going to be something about us as a couple and the moment one of us was unhappy then it stopped as our relationship is the most important thing.
Then I brought up this woman. He laughed a bit when I said something was bothering me and that it was her but I said no this is important and upsetting me so he said sorry ok go on... anyway I laid out everything that had bothered me (the names she calls him, the meetings at our house etc).
He said he can understand why an outsider would say this is dodgy don't let your husband see this woman but if they met her they would say there is nothing to worry about (I see where he is coming from here as she is a nice person it is just the weird things that put me off her a bit).
Regarding the names he said she calls all men at the group by these pet names and then he said and how is it different from the names he calls people (he calls everyone names like sweetie, angel, gorgeous - men and woman, young and old). He also said he can't really say to her no my names not gorgeous its Bob (not his real name) as it sounds weird and rude so he is not sure how to stop that happening if it is upsetting me.
Regarding the meetings he has at our house he said they happened for the first 4 or 5 weeks he started doing the job of all the reports and she helped him but she doesn't anymore. He said she goes out for coffee with several different men from the group it is not always him (it was at first as he introduced her to the group when he moved from his old one and he was the only person she knew there).
About the affair gossip he says that has died down now that everyone knows her better and sees she is like that with all the men. He did say he wasn't offended when they were saying it and it was an ego-boost that people would think that (he has always been down on his looks) but I said that was stupid and that it was offensive to me. He said ok fair enough.
Regarding the 'lie' about going to her house he says I misunderstood that and he wasn't lying.
He said in the end it all comes down to perception - he could see how someone who wasn't here in our situation would see it but that they would be wrong. From my point of view he is sorry that it upsets me but he cannot stop seeing her completely as the group gets him a large amount of work. He said he is not encouraging her but if she did happen to come on to him (he is adamant this would never happen) then he said he would immediately cut her off. In the end he said there is nothing to worry about. He said he is aware of what kind of woman she is and that she likes attention (particularly from men) but he does not flirt with her. He also said he thinks that after her divorce she is just letting her hair down and acting like a teenager after being in an unhappy relationship so long.
I am glad I had this chat with him as I think for some of it my imagination was running away with me. The fact is he has never kept this woman secret from me in fact he always invites me to the events where she will be too except on the days he goes to the meetings (he has actually invited me to go to one of the meetings).
I am still going to keep my eye on her when she is about as I don't trust her one bit - although I do believe she is more about liking the men fancying her rather than being a predator.
However I do trust my DH. I'm sure some of you will think I am naive but I know him and know what sort of man he is. We were best friends for 6 years before we got together and I trust him implicitly. Yes he was out of order for being pleased by the ego boost of the affair talk but I honestly think that is the only wrong he has done here. Either way I will be on the watch for anymore dodgy behaviour just in case.
Thanks very much for all your support.