Your reason for not saying no seems to be that they didn't ask. That's even worse - if they're telling you what will happen in your own home, you tell them no!
For some reason I still can't fathom, my mum used to do similar. We had a dog who was great with people, would attack other dogs (she was an older rescue, maybe lack of socialisation from previous owner). My mum absolutely delighted in saying she'd invited dog owners to stay with us with their dogs (bearing in my mind DP and I are adults with our own house), as well as friends of hers we literally did not know from Adam. We nipped it in the bud with blunt responses and it quickly stopped.
"I've told your cousin to bring her dogs to your house and that her and dog are welcome to stay with you whenever they like!"
"Oops, well you know our dog will attack hers, and it's our house to invite people, so that can't happen. Please do let her know she's absolutely fine to arrange a visit directly with us, without the dog, if it suits everyone, when you call her back to explain."
Just something like that, it's not rude to stand up to people who dictate to you, especially if, as in my case, it's done to prove a point or try to wind you up. My mum has a history of controlling and deliberately provocative behaviour though, so maybe a little gentler if these people are otherwise fine!