What a sad situation for you all. I take it, it is the maternal grandparents?
I wouldn't be happy about this either, it would grind me down. Your SD is not having a good relationship with her Dad if it is always on 'Mums/Mums Grandparents turf' where her Mum can turn up at any time. That isn't in any way a basis for building a relationship with his daughter. It is just perpetuating what seems to have started off horribly in a cycle of chaos and now control.
I would also not be happy at all about your daughter, just your daughter going there in that kind of atmosphere. What are they doing? Is he just hanging around doting on his older daughter? What is your daughter learning from this? What is your son? This is directly affecting your family.
Does your DP listen to you at all? Why on earth is there not even some time with his daughter at your house or he takes her out? At present it sounds like it would be better having less time, but his 'own' time with his daughter, taking her out for a meal and the cinema, anything. A relationship is not one so totally dictated like this, he really doesn't have what he thinks he has.
However, will he stand up and do this? Quite possibly not, and he may have to find his own way. However if I were you I would absolutely not be taking part in this ridiculous charade of letting your daughter go with him. If SD wants to see her sister, then she has to see him with her brother, that is appalling behaviour towards a completely innocent boy and I would think it is totally in THEIR best interests to protect your kids from being essentially manipulated like this. I hope SD reaches out at some point later in her life to you all, as apart from her grandparents, you are a stable part of her life. But not like this!