I think maintaining a positive relationship with your DS's family is an excellent thing, and better for her than handing her over for an access visit to her GPS and aunts etc,
The best outcome would be that your bf also become an accepted part of the scene. Do your exILs know about him?
In what way is it a 'red flag' if someone stays part of a wider family group of which their child is a member? In what way is that a threat to or betrayal to a new partner unless it is about jealousy and territory? I can see new partners might have a bit if a question about it but if there is no agenda of unfinished business then it is the new partner that needs to come to terms with what is essentially their problem.
'No longer part if their family'! What is this? Tribal loyalty and a woman has to be the property / member of one tribe or the other ? Families are more complex and more sophisticated in this century and you sound as if you are doing a good job, OP.
I would stay with a man who was open and communicative about any discomfort he feels, and would talk it through, offer to introduce him etc, and presumably if you have a birthday tea, for example, for your dd, you will invite her grandparents? But if he tried to make me dismantle a social network that works well for my dd, or interfere with that in any way, I would not be having it.
Good luck, I hope he calms down.