You could look up the sort of conditions imposed under a Considerate Builders Scheme. (Westminster have one, but there will be others.)
This will set out things like the hours builders might start or finish, hours they might work during the weekend, when deliveries might arrive and so on. No on site radios if summer.
I allowed a neighbour to build along a boundary in a rental property, on condition he allowed me to use the same wall should I ever extend. The council imposed a considerate builders scheme as part of their planning permission.
However he moved out and left the builders to it. They took down the whole fence rather than the portion they needed to, started using my garden as a storage area, leered at the tenants if they attempted to use the garden, and even used to peer in the windows. And as they started to over run they worked through weekends, starting early. Even more frustrating there was no apology at the end of it, or a bottle of wine. Really grating given he worked for a wine merchants and had a whole house full of samples.
Luckily the tenants did not blame me, though I did have to then buy them the gift he should have given them. (Not from his firm obviously!)
So speak to neighbours, apologise in advance, don't assume access but ask politely and ideally offer them something in exchange like new fencing, and explain the guidelines under which the builders will operate and how they will be enforced, listening to any specific issues/problems your neighbours might have. Then a thank you afterwards as a way of repairing relations. Having a build next door is irritating however much you try to avoid problems.