Fingers crossed they are just clueless and extremely horrified about being so short sighted etc
And you saying you hope you made a mistake, surely they still should have notified you about needing access to your garden etc. No way is this something you've got wrong, irregardless of planning permission and a grade listed building, it's not their land!