I really can't believe how passive you are being! You don't even sound interested in discovering exactly WHY your DD doesn't want to go to his, and indeed has said she wishes she didn't even have a Dad!! Surely that is a potentially huge red flag!!?
Your DD is being very clear here trying to tell you that something is not right but you really don't seem to be listening and, whatever you say, you don't seem to have done any digging or advocated for her so far.
If I couldn't even have the address my children would be staying at then they absolutely wouldn't be going there. At all. Yes, you can put a stop to this if it currently isn't court ordered, and I have the definite impression that it isn't.
You NEED to know where they are otherwise what if a serious issue occured? How would you get to them? You also need to know what the living conditions are really like. If they are as inadequate as you say then the children should not be going there, especially not for overnights.
Stop giving a shit what others like your ex-PILs think about you and start giving more of a shit about what is happening to your poor child/children here. They only have you to advocate properly for them and so far all you seem to be saying is "but I can't do that!" Which is very wet to be honest. Yes, you absolutely CAN do this.
Your children are potentially being seriously damaged here. Especially the child who wishes she didn't have a Dad!