OK, left field interpretation (but it's something that happened to me with an ex). Is there any possibility he is interested in swinging, something like that? And when you asked if he had pulled, he kind of thought, ooh, green light, she likes the idea of imagining me with another woman, I will play into this fantasy a bit here.....
I think it's something people can find hard to bring up in relationships, for fear of their partner reacting in horror, so tho it would be better to just be up front, they often look for little clues instead.
I'd have it out with him. Ask him if he fantasies, maybe about being with other people and telling you/you watching/,being involved even. And was that what was going on?
If so, you can just tell him either to drop it (or join in with the fantasy, whatever your preference is).
If it's not that then he's an utterly disrespectful dick and I would kick him to the curb!