I had been dating a woman for a few weeks when we got into the 'previous relationships' conversation and I told her that I was still friends with two or three of the women I'd been out with, one of whom was the mother of my son. (I was fortyish at the time.)
She said that she could see that I'd have contact with my kid's mum, but she wasn't happy about me seeing any of the other women.
I said, "Why? They're friends. I don't think you can tell me to dump my friends."
He previous relationship, it turned out, had ended because the bloke was having a thing with an ex. So one could see how she was a bit sensitive to the possibility.
She suggested a compromise, and it was very specific. I could see one ex per month. For lunch, not for dinner.
I said, "That's nuts. I mean, if I was going to have sex with them, I don't think I'd feel it was impossible before teatime, do you?"
She was utterly convinced that this was an equable arrangement, and that in the interests of our future, I should agree to it.
I pointed out that she was in effect asking me to choose between herself - who I'd known for six weeks - and friends I'd known for years, and in once case since I was seventeen. How, I asked her, do you think that's going to go?
It went as you might expect.
However, what she was afraid of really, wasn't the possibility of sex, but that I had a history with those women that she couldn't match - memories, shared experience, emotional understanding. And of course that was true.
The issue, I think, is whether the OP thinks that that connection with the ex is still sufficiently strong that it'll get in the way of the formation of a strong connection in the new relationship.
But if you demand a stop to it, and it's not still a romantic thing, be aware that you're telling him how to conduct his friendships. Not many people will stand for that.