Drop him. I had the same with ex. He maintained contact with his exes including one he still stayed over with, if he was in her neck of the woods. I was a horrible person when I was with him. Wound up like a spring.
I came to realise, that's what he wanted. To de-stabilise relationship by keeping contact with women he'd had a sexual relationship with. & Making sure I knew it. Far too many of our disputes were about exes. Even if I didn't raise the matter, he'd find a way to mention an ex then that was it, evening spent arguing. I bet your man mentions exes all the time to you OP.
I also felt it was so unkind to his exes...as if he was ensuring he was never completely out of their lives so as to prevent them from moving on properly, finding relationships of their own. I think men like this deep down neither like not respect women. It's the ego stroking that's important
I left him to his pseudo-harem games eventually. I have male friends current OH has female friends..it's normal. No feeling of others being part of our personal relationship dynamic.
I'd drop the man, in your shoes. He is bad for your emotional well-being. When you leave him be aware he'll try to make you one of the exes he dangles too, if you maintain contact.
It's up to you to realise you'll likely be upset without him but you aren't going to die for lack of him.