This will all come to head, and probably sooner rather than later.
The next time he goes off to meet them.
Either you're able to join them, or you're not. And if you're not and you accept it, well, that's that.
The ex's family is almost a red herring. The issue is actually how much this man cares about, and prioritises, you and your feelings.
It's actually OK for him to fundamentally feel as if his ex is more important (his feelings are his feelings after all). If you lay down the law to him, it's only going to lead to resentment on his part, driving a wedge into your relationship anyway. Just like the current situation cause resentment for you.
This is why it's inevitable that it will all come to a head.
So then you decide, like his previous girlfriend, whether it's a deal-breaker. It certainly would be for the majority of people.