Surely the builders will have spoken to them, they will already know this is the end of it, Don't go & see them alone, but do say its a private garden, they can't use it, & its a listed building & hope they have done the paperwork. If not, they had better put it back exactly as before pronto,
no need for signs, for fences, or if not the council are being called.
It may be a genuine mistake. you will have to live near these people for a long time
hope they haven't already promised a puppy to their DC