In a way, it's quite lucky that the building is listed as it means it is out of your hands, and you don't have all the legal shit of enforcing them to put everything back - the council will do that. So as much as possible, leave it to the council.
They will need access to your garden to put it right. Ask that they give you notice in writing. When you give your consent in writing, also make it clear that you want your garden putting back as it was (take photos), not infringed on anymore than is necessary, and specify when the work can take place. It is reasonable to specify normal working hours (i.e. not evenings and weekends).
Here is some information on the Party Wall Act
Bit pernickety, but if I were you, I would not let them even use your garden to come and talk to you. Point out that they do not have access to your garden, and ask them to come to the front door.