Not sure that I would find it do much a problem if my DH agreed that it was a bit annoying. They are nice people and I know they are just being friendly , but it is too much. If I go out, I come home to find the dog gone - they have taken her to theirs - even if I have just popped out for 2 seconds.
They will just let themselves in whenever, and just say they want a chat.