As others have said, it is a common fantasy but, on reflection, it seems to me that this has gone way beyond a fantasy and has become something of a lifestyle choice for your h.
You've said that you overcame your sexual repression with the encouragement of your h. However, it seems that his encouragement was more directed towards his own sexual needs and gratification than yours because he, effectively, turned you in a 'whore' (please understand that I'm not using that word judgementally or descriptively) to the extent that you fulfilled his fantasy of watching 'his woman', 'his slut', 'his creation' having sex with other men in real life.
Rightly, you refused to let the 'fantasy' spill over into your everyday life by coming down firmly in respect of the dress and other incidents, otherwise I suspect that he'd have you dressing to please him and entertaining other men whenever he wanted to play the voyeur.
The approach you should take now is to tell him that you've both taken his 'slut fantasy' (for want of a better term) as far it can go; you've both been there, done that, got the t-shirt, and now it's time to move on to a more a mature and exclusive relationship in which your joint sexual pleasure does not rely on other real life participants or fantasies about other wo/men joining the party, so to speak.
The problem is that this 'fantasy' has most probably been a part of his sexual psyche for a very long time - perhaps from puberty - and, as you've said, it's become hardwired into his brain.
As your h appears to be sexually dependant on this particular sexual mindset, he may find it difficult to become fully aroused or achieve satisfaction without resorting to it in which case, as mmaq suggested earlier, you should make it clear to him that 'he can be thinking what he likes, but he doesn't need to mention it' and that, unless you are in the mood to pull it out of the box, you should keep your individual fantasies non-verbal when you are engaging in sexual intercourse.
As it will be hard for him to break the sexual habits of almost a lifetime, perhaps you could reassure him that you'll indulge him on high days and holidays, or may be once a month if you feel so inclined, but in return you expect him to indulge you quid pro quo in a fantasy of your choosing.