OP, I understand completely.
Almost the same situation. Not exes per se but ex friends with benefits. All in the same large circle of friends (male and female).
I tried the whole being friends with them thing and it didn't work. Not even really because he'd slept with them, they were all girly girls and I mostly felt like an alien when I hung out with them. And they all felt like they needed to let me know they had some sort of claim on him. Gross.
Now an occasional like on Facebook is as far as it goes etc and he has naturally distanced away from that group gradually over the years.
It is all horribly immature and not "me" at all - but I love him and we all have a past so what can you do?
He finally realised how it made me feel quite early on (a year or so ago) when I asked him how he would feel if I was going to a party with Jim, Dave, Oliver, Mike and Ben (not real names 
) and it was like it all clicked and he realised he wouldn't like it.
As I said, thankfully he had matured a lot now and doesn't hang out with them all that often.