They now have started inviting him for dinner, to hang out etc, which he is very happy to do. I feel really uncomfortable about it - he behaved so badly during our relationship
I'm sorry you're going through this.
I mean do they know he was bad? or is he like a typical narc, where everyone thinks he's amazing and you're the bad one for breaking the "family" apart? Regardless, it doesn't matter. You can't change their perception right now, you'd look vindictive.
In this scenario, I'm afraid I'd have to set my own boundaries and be prepared to lose some "family" or go NC/LC with them, if that means not attending events if he has been invited, or will be there etc. You do what you need to do for your own comfort. If they want to maintain a relationship, that is their decision, you can't dictate their decision, but you can, as a result of their decision make your own decision to not see them (when he is there or at all, depending on your stance).
It's painful because you assumed that family would "stick" by you, take your "side" - that's not always the case.
It doesn't matter, you can't control how others react, you can only control your reactions to their actions.
Let them Theory - Mel Robbins.