I don't tolerate this kind of thing so I would not be in a relationship with someone like him for five minutes.
Being married for many years now, it's staggering how much my husband and I invest in each other, in the past and now. I am quite happy to do it because I get the same treatment in return. I would not be at all happy with the sacrifices and etc. if I had to wonder why he was taking trips alone with his old girlfriend. We don't do nor tolerate any hint of this kind of thing.
One of our "rules" is that he and I (and our kids, to some extent) are in the center of "our world." Nobody else is allowed into that inner circle. If I don't like someone, they're out and if he doesn't like someone, they're out, because we are always number one to each other. That makes it all real simple, to us.
Aside from all that, no, we don't do opposite sex friends anyway unless there's a reason for an exception AND it's okay with our partner. For ex. I have a longterm gay male friend from childhood. My husband happily allows it and has become friends with him too. But if he didn't, that would be it, because husband always comes first to me and vice versa.
I think your boyfriend is not putting your position with him nor your feelings first and that's definitely a problem. My husband and I would not have to be told not to pal around with our exes or others of the opposite se. We never would because that is allowing an opposite sex person in too close, into our partner's space and position. We would never want to hurt each other or make each other worry and doubt like that.
Also, to be honest, it sounds like obvious BS and my guess is he's having affairs. But that doesn't need to be proven for you to demand better treatment or leave. But if you even have to tell him not to do this in the first place, well that is not looking good to me. I'd say we know that you are in a committed relationship but it's doubtful he is. I'd toss him out before you get in any deeper and go find someone who feels far more honored to have you by their side, and acts like it. :)